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Abstract 


This  report  describes  RMRATE,  a  computer  program  for  analyzing 
rating  judgments.  RMRATE  scales  ratings  using  several  scaling  pro- 
cedures, and  compares  the  resulting  scale  values.  The  scaling  proce- 
dures include  the  median  and  simple  mean,  standardized  values,  scale 
values  based  on  Thurstone's  Law  of  Categorical  Judgment,  and 
regression-based  values.  RMRATE  also  computes  reliability  statistics 
and  analyzes  the  ratings  using  principal  component  analysis  tech- 
niques to  assess  variation  among  judges.  RMRATE  should  be  useful 
to  practitioners  needing  to  summarize  or  analyze  rating  data,  and  to 
researchers  interested  in  comparing  and  evaluating  alternative  scaling 
methods. 
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Analysis  of  Ratings:  A  Guide  to  RMRATE 

Thomas  C.  Brown,  Terry  C.  Daniel,  Herbert  W.  Schroeder,  and  Glen  E.  Brink 


INTRODUCTION 

Rating  scales  provide  an  efficient  and  widely  used 
means  of  recording  judgments  about  many  kinds  of 
stimuli  (Nunnally  1978).  This  report  describes  RMRATE, 
a  computer  program  for  analyzing  rating  data. 

RMRATE  is  a  batch  program  written  in  FORTRAN  77 
for  use  with  an  IBM-compatible  personal  computer.  The 
program  is  designed  to  (1)  scale  rating  data  using  a  bat- 
tery of  scaling  procedures;  (2)  compare  the  scale  values 
obtained  by  use  of  these  procedures;  (3)  evaluate,  to  a 
limited  extent,  whether  the  assumptions  of  the  scaling 
procedures  are  tenable;  (4)  determine  the  reliability  of 
the  ratings;  and  (5)  evaluate  variations  among  raters. 

The  scaling  procedures  of  RMRATE  are  described  in 
detail  in  a  companion  paper  by  Brown  and  Daniel  (1990). 
That  paper  and  the  RMRATE  computer  program  are  an 
outgrowth  of  an  effort  that  began  in  the  early  1970s  to 
better  understand  the  effects  of  management  on  the 
scenic  beauty  of  forest  environments.  An  important 
report  by  Daniel  and  Boster  (1976)  introduced  the  Scenic 
Beauty  Estimation  method  and  made  available  a  com- 
puter program  for  scaling  rating  data  to  "SBEs." 
RMRATE  includes  SBEs  among  the  scale  values  it  esti- 
mates from  rating  data,  and  provides  a  more  straight- 
forward output  of  the  results  than  the  earlier  program. 
In  addition,  RMRATE  computes  median  and  mean 
ratings,  standardized  scores,  and  a  new  scale  based  on 
a  least  squares  analysis  of  the  ratings. 

While  scenic  beauty  has  been  the  focus  of  the  work 
that  led  up  to  this  report,  the  utility  of  RMRATE  is  cer- 
tainly not  limited  to  measurement  of  scenic  beauty. 
Rather,  RMRATE  should  be  of  interest  to  anyone  need- 
ing to  analyze  ratings,  no  matter  what  the  stimuli. 

Use  of  RMRATE  requires  an  understanding  of  psycho- 
logical scaling  and  experimental  design.  Readers  with 
no  prior  knowledge  of  scaling  methods  should  consult 
our  companion  paper  (Brown  and  Daniel  1990)  and  a 
basic  text  on  the  subject,  such  as  Nunnally  (1978)  or  Tor- 
gerson  (1958).  We  suggest  that  the  reader  desiring  an 
explicit  treatment  of  experimental  design  also  consult 
a  basic  text,  such  as  Cochran  and  Cox  (1957)  or  Camp- 
bell and  Stanley  (1963). 

Brief  Description  of  RMRATE 

RMRATE  accepts  from  5-point  to  10-point  rating  scale 
data.  The  program  analyzes  rating  responses  by 
"observers"  about  "stimuli."  The  stimuli  may  be 
grouped  into  "conditions."  For  example,  in  a  study  of 
forest  scenic  beauty,  a  condition  might  be  a  timber  stand, 
with  the  stimuli  being  individual  scenes  within  the 
stand.  Scaled  responses  about  the  stimuli  are  provided 
for  each  stimulus  and,  if  applicable,  each  condition. 


If  the  set  of  stimuli  to  be  rated  is  too  large  to  be  rated 
in  one  session,  and  it  is  necessary  to  obtain  ratings  in 
two  or  more  sessions  and  then  combine  the  ratings  from 
the  different  sessions,  a  "baseline"  condition  may  be 
specified,  containing  stimuli  that  are  rated  in  each  ses- 
sion. The  baseline  ratings  provide  a  basis  for  determin- 
ing the  comparability  of  the  ratings  obtained  in  the 
different  sessions  and  may  allow  computation  of  scale 
values  expressed  relative  to  responses  for  the  baseline 
stimuli.  See  Brown  and  Daniel  (1990)  for  more  on  base- 
line adjustments. 

RMRATE  provides  two  basic  types  of  information, 
scale  values  and  statistics.  Scale  values  are  computed 
from  the  raw  ratings  according  to  several  scaling  proce- 
dures, from  simple  means  to  standardized  (Z)  scores  to 
scales  based  on  Thurstone's  Law  of  Categorical  Judgment 
(i.e.,  SBEs).  Statistical  information  about  the  ratings 
includes:  (1)  indications  of  the  distribution  of  the  rat- 
ings (e.g.,  range,  standard  deviation,  skewness),  (2)  indi- 
cations of  the  correspondence  among  observers  (e.g., 
reliability  coefficients),  and  (3)  analysis  of  the  ratings 
using  principal  component  analysis. 

RMRATE  first  reproduces  the  full  data  set  (the  raw 
ratings)  and  lists  the  mean  and  median  ratings  of  each 
observer  and  for  each  stimulus,  the  standard  deviation 
and  range  of  ratings  by  each  observer  and  for  each  stimu- 
lus, the  skewness  of  each  observer's  ratings,  and  the 
correlation  of  each  observer's  ratings  with  the  mean 
ratings  across  all  other  observers. 

The  user  has  the  option  to  remove  observers  or  stimuli 
from  the  analysis  of  the  ratings  according  to  several 
criteria.  For  example,  an  observer  may  be  removed  if  he 
or  she  failed  to  rate  more  than  some  specified  number 
of  stimuli.  If  any  observers  or  stimuli  are  removed,  the 
original  table  may  be  reproduced  with  only  the  remain- 
ing observers  and  stimuli  included. 

RMRATE  then  reports  reliabilility  statistics  indicating 
the  expected  correlation  of  ratings  obtained  from  any  two 
observers,  and  of  group  mean  ratings  obtained  from  any 
two  observer  groups,  drawn  from  the  same  population 
as  the  sample  group.  The  mean  of  the  correlations  of 
each  observer's  ratings  with  the  mean  ratings  of  the  rest 
of  the  group  of  observers  in  the  session  is  also  listed. 

Next,  the  program  lists  the  scale  values  for  each  stimu- 
lus and  each  condition  according  to  10  procedures  for 
scaling  rating  data.  These  procedures  include  the 
median,  mean,  standardized  (Z)  score,  and  least  squares 
rating,  plus  several  baseline-adjusted  procedures  includ- 
ing the  by-stimulus  SBE.  Some  of  the  procedures  attempt 
to  compensate  or  adjust  for  potential  problems  with 
rating  scale  judgments,  as  described  by  Brown  and  Daniel 
(1990).  RMRATE  does  not  attempt  to  determine  the  rela- 
tive merit  of  these  procedures;  rather,  the  procedures  are 
all  included  in  RMRATE,  along  with  statistics  indicat- 
ing the  extent  to  which  some  of  the  assumptions 
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of  the  scaling  procedures  appear  tenable.  The  statistics 
include  standard  deviation,  skewness,  kurtosis,  and  the 
Anderson-Darling  test  for  goodness-of-fit  to  a  normal  dis- 
tribution. The  aim  is  to  provide  the  user  with  the  means 
to  evaluate  the  utility  of  the  various  scaling  procedures 
for  a  given  application. 

RMRATE  then  reports  two  indications  of  the  distri- 
bution of  ratings:  the  proportion  of  observers  using  each 
possible  range  of  the  rating  scale  and,  for  each  stimulus 
and  condition,  the  proportion  of  responses  in  each  of 
the  possible  rating  categories. 

Next,  RMRATE  compares  the  results  of  the  different 
scaling  procedures  by  listing  correlation  coefficients  and 
producing  plots  of  the  scale  values  for  pairs  of  the  scal- 
ing procedures.  This  is  done  across  stimuli  and  across 
conditions.  For  example,  the  correlation  and  plot  of  the 
mean  ratings  with  the  SBEs  is  listed,  first  across  the 
stimuli  and  then,  if  applicable,  across  the  conditions. 
The  correlation  coefficients  indicate  the  degree  of  linear 
relationship  among  the  scale  values,  and  the  plots  help 
indicate  whether  a  nonlinear  relationship  exists. 

Next,  tables  are  provided  of  Z-scores  and  least  squares 
scores  for  each  original  rating  (for  each  rating  by  each 
observer).  These  tables  may  be  useful  in  further  analyses 
performed  using  other  software. 

Next,  a  principal  component  analysis  of  the  ratings 
is  provided,  complete  with  tables  of  eigenvalues,  com- 
ponent loadings,  component  scores,  and  plots  of  the 
loadings  and  scores  for  selected  pairs  of  components. 
This  analysis  can  be  used  to  assess  the  homogeneity  of 
a  group's  ratings,  and  to  reveal  whether  subgroups  of 
observers  rated  the  stimuli  in  consistently  different  ways. 

Finally,  a  table  lists  each  observer's  median  and  mean 
rating  and  by-observer  SBEs  for  each  condition,  plus 
indications  of  the  extent  to  which  each  observer's  rat- 
ings are  normally  distributed. 

An  additional  feature  of  RMRATE  is  the  option  to 
produce  more  machine-readable  output  of  the  scale 
values  than  is  available  in  the  standard  output  tables. 
This  output  format  is  useful  if  the  scale  values  are  to  be 
used  as  data  in  further  analyses. 


Obtaining  a  Copy  of  RMRATE 

An  executable  copy  of  RMRATE  can  be  obtained  by 
writing  to  the  first  author,  and  enclosing  a  blank  format- 
ted 5V4-inch  floppy  or  3V2-inch  diskette.  That  diskette 
will  be  loaded  with  four  files:  with8087.exe,  no8087.exe, 
sample.exe,  and  read. me.  The  first  two  files  install  the 
executable  version  of  RMRATE  appropriate  to  the  com- 
puter's memory.  The  first  requires  a  math  coprocessor, 
and  the  second  does  not.  The  third  file,  sample.exe, 
installs  five  files:  SAMPLE.DAT,  a  sample  data  set 
identical  to  that  used  in  this  paper  to  illustrate  the  pro- 
gram; SAMPLE. RUN,  sample  input  for  the  data  set; 
SAMPLE. OUT  and  SAMPLE. SAV,  the  output  files  that 
the  sample  input  and  data  produce;  and  SAMPLE.BAT, 
a  batch  file  for  recreating  SAMPLE. OUT  and  SAM- 
PLE.SAV  given  SAMPLE. RUN  and  SAMPLE.DAT. 
Read. me  explains  how  to  install  the  other  files. 


With  modification,  RMRATE  can  be  uploaded  to  run 
on  a  mainframe  computer.  All  machine-dependent  code 
is  contained  in  one  module  of  the  program,  which  is 
easily  modified  by  a  FORTRAN  programmer.  Persons 
wishing  to  upload  RMRATE  to  a  mainframe  can  request 
the  source  code,  again  by  sending  the  first  author  a  blank 
formatted  diskette. 


STATISTICS  OF  RMRATE 

RMRATE  computes  two  types  of  information  from 
ratings:  scale  values  and  statistics.  The  scaling  pro- 
cedures are  described  in  detail  by  Brown  and  Daniel 
(1990),  and  the  statistics  are  described  here.  Three  types 
of  statistical  information  about  a  set  of  ratings  are  pro- 
vided: (1)  indications  of  the  distribution  of  the  ratings; 
(2)  indications  of  the  correspondence  among  observers; 
and  (3)  analysis  of  the  ratings  using  principal  compo- 
nent analysis  to  reveal  whether  subgroups  of  observers 
rated  the  stimuli  in  consistently  different  ways. 

Distribution  of  Ratings 

Measures  that  indicate  the  distribution  of  ratings  by 
observers  and  by  stimuli  are  useful  in  considering 
whether  assumptions  of  the  scaling  procedures  are  ten- 
able. The  following  measures  reported  by  RMRATE  indi- 
cate the  distribution  of  each  observer's  ratings  (the  dis- 
tribution of  the  ratings  assigned  to  the  various  stimuli 
by  a  given  observer): 

1.  Range 

2.  Standard  deviation 

3.  Skewness 

4.  The  proportion  of  observers  that  used  each  possi- 
ble range  of  the  rating  scale. 

The  following  measures  indicate  the  distribution  of 
ratings  assigned  by  an  observer  group  to  each  stimulus: 

1.  Range 

2.  Standard  deviation 

3.  Proportion  of  ratings  in  each  rating  category 

4.  Skewness 

5.  Kurtosis 

6.  Anderson-Darling  statistic. 
Range 

The  range  indicates  the  distance  from  the  lowest  rating 
to  the  highest  rating  that  a  given  observer  assigned  to 
the  stimuli,  or  that  the  observers  assigned  to  a  given 
stimulus.  For  example,  if  across  all  stimuli  an  observer 
only  assigned  ratings  of  6,  7,  and  8,  the  range  is  2;  or, 
if  an  observer  only  assigned  ratings  of  2  and  7,  the  range 
is  5.  Given  a  10-point  rating  scale,  the  range  may  vary 
from  0  to  9. 

Normality 

The  skewness  and  kurtosis  measures  and  the 
Anderson-Darling  statistic  indicate  the  degree  to  which 
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sets  of  ratings  are  normally  distributed.  The  finding  that 
ratings  are  normally  distributed  many  lend  support  for 
acceptance  of  the  assumptions  of  certain  scaling  pro- 
cedures. In  terms  of  stimuli,  skewness  (Snedecor  and 
Cochran  1980)  of  the  ratings  is  computed  as: 


(Rii-MRi)]5 


[l] 


where 
Sk;  = 


mr;  = 


skewness  of  stimulus  i 
rating  of  stimulus  i  assigned  by  observer  j 
mean  rating  assigned  to  stimulus  i 
S;      =  standard  deviation  of  ratings  assigned  to 

stimulus  i 
n      =  number  of  observers. 

Skewness  measures  deviations  from  symmetry.  Sk  will 
be  zero  when  the  distribution  is  completely  symmetric 
(e.g.,  a  bell-shaped  curve).  A  positive  value  indicates  that 
the  ratings  are  clustered  more  to  the  left  of  the  mean,  with 
most  of  the  extreme  values  to  the  right.  A  negative  value 
indicates  clustering  to  the  right.  The  measure,  of  course, 
assumes  ratings  may  be  used  as  an  interval-level  metric. 

Kurtosis  (Snedecor  and  Cochran  1980)  of  the  ratings 
for  a  stimulus  is  computed  as: 


Ki=  { 


n 

i  =  i 


[^(Rii-MRi)]4}  -3 


[2] 


where 


Rij  = 
MRj  = 


kurtosis  of  stimulus  i 
rating  of  stimulus  i  assigned  by  observer  j 
mean  rating  assigned  to  stimulus  i 
standard  deviation  of  ratings  assigned  to  stimu- 
lus i 

n     =  number  of  observers. 

Kurtosis  measures  the  relative  peakedness  or  flatness 
of  the  curve  defined  by  the  distribution  of  ratings.  A 
normal  distribution  will  have  a  kurtosis  of  zero.  If  the 
kurtosis  is  positive,  the  distribution  is  more  peaked  than 
is  the  case  with  a  normal  distribution  of  the  same  stand- 
ard deviation;  if  the  kurtosis  is  negative,  the  distribu- 
tion is  more  flat  than  that  of  a  normal  distribution  of  the 
same  standard  deviation. 

The  Anderson-Darling  statistic  can  be  used  to  test  for 
goodness-of-fit  to  a  normal  distribution.  It  is  one  of  a 
class  of  EDF  (empirical  distribution  function)  statistics 
that  includes  the  Kolmogorov  statistic,  the  Watson  statis- 
tic, and  others  (Stephens  1974).  Where  the  population 
mean  and  standard  deviation  are  unknown,  and  must 
be  estimated  from  the  sample,  Stephens'  (1974)  case  3 
is  appropriate.  The  Anderson-Darling  (A2)  case  3  statis- 
tic for  the  ratings  for  a  given  stimulus  is  computed  as: 

A2  =-{E(2j-l)[lnzj  +  ln(l-zn  +  1_j)]}/n-n  [3] 

j  =  i 

where  the  ratings  are  ordered  in  ascending  order  and 

Zj     =  cumulative  probability  of  a  standard  normal 
distribution  of  the  value  w; 


Wj    =  (R:  -  MR)/SDR,  the  standardized  score  for  Rj 

R:     =  rating  by  observer  j 

MR  =  mean  rating  for  the  stimulus 

SDR  =  standard  deviation  of  the  ratings  for  the 

stimulus 
n     =  number  of  observers. 

The  test  for  normality  can  be  simplified  by  adjusting 
A2  for  sample  size  using  the  following  relationship 
(Stephens  1974): 

A2*  =  A2  (1  +  4  /  n  -  25  /  n2)  [4] 

where 

A2*  =  modified  Anderson-Darling  case  3  statistic 
n    =  sample  size  (number  of  observers). 

This  modified  statistic  is  then  compared  with  the  signif- 
icance point  of  the  chosen  significance  level  (one-tailed 
test)  (Stephens  1974:  table  1.3): 


Significance 
level  (%) 

15 
10 

5 

2.5 
1 


Significance 
point 

0.576 
0.656 
0.787 
0.918 
1.092 


The  null  hypothesis  that  the  population  from  which 
the  sample  of  ratings  was  drawn  is  normally  distributed 
is  rejected  if  A2*  exceeds  the  cutoff  point  for  the  selected 
significance  level.  For  example,  if  a  sample  of  ratings 
yields  an  A2*  of  0.780  for  a  given  stimulus,  we  would 
conclude  that  the  distribution  of  ratings  does  not  differ 
significantly  from  a  normal  distribution,  having  sub- 
jected our  test  to  a  5%  chance  of  a  type  I  error  (of  reject- 
ing the  null  hypothesis  when  it  is  true). 


Correspondence  Among  Observers 

Statistics  of  correspondence  indicate  how  much  con- 
fidence one  should  place  in  the  scores  if  they  are  to  be 
used  for  decisionmaking  purposes.  As  described  by  Ebel 
(1951)  and  Tinsley  and  Weiss  (1975),  upon  whose  work 
much  of  this  discussion  is  based,  such  statistics  are  eas- 
ily computed  using  correlation  computations  or  the 
results  of  a  standard  analysis  of  variance  (ANOVA)  of 
the  ratings.  Strictly  speaking,  these  statistics  assume 
interval  properties  for  the  ratings,  but  they  have  been 
used  extensively  for  evaluating  rating  scale  responses. 


Reliability  and  Agreement 

The  literature  contains  a  confusing  mixture  of  terms 
for  the  different  measures  of  correspondence  among 
observers.  We  have  adopted  Tinsley  and  Weiss'  (1975) 
distinction  between  "reliability,"  which  focuses  on  rel- 
ative differences  among  observers'  ratings  (measured 
by  the  correlation  coefficient),  and  "agreement,"  which 
emphasizes  absolute  differences  among  observers' 
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ratings  (measured  by  the  differences  between  actual 
ratings). 

The  choice  between  reliability  and  agreement  depends 
on  whether  or  not  ratings  from  one  observer  or  group 
are  going  to  be  compared  with  those  of  another  observer 
or  group  in  the  course  of  decisionmaking.  For  example, 
considering  group  mean  ratings,  the  distinction  is 
between  decisions  that  will  be  based  on  one  set  of  aver- 
aged ratings  and  decisions  requiring  absolute  compari- 
sons of  two  or  more  sets  of  mean  ratings.  If  such 
comparisons  are  necessary  for  decisionmaking,  the  ab- 
solute values  of  the  ratings  are  important  (i.e.,  a  mean 
rating  of  "5"  by  one  observer  group  must  be  compara- 
ble to  a  "5"  by  another  observer  group).  In  this  case, 
an  "agreement"  statistic  would  be  needed.  If  such  com- 
parisons between  groups  are  not  necessary  for  decision- 
making, then  the  absolute  values  of  the  ratings  is  not 
important;  here,  only  the  relative  differences  among 
stimuli  matter.  In  this  case,  a  "reliability"  statistic  is 
sufficient.  Our  position  is  that  absolute  comparisons  of 
mean  ratings  obtained  from  two  separate  observer 
groups,  or  of  individual  ratings  obtained  from  separate 
observers,  are  generally  inappropriate  and  that,  if  com- 
parisons between  groups  are  necessary,  it  is  best  to  use 
a  baseline  procedure  and  scale  the  ratings  to  some  meas- 
ure, such  as  the  SBE*,  that  utilizes  the  baseline  ratings 
to  adjust  for  differences  between  groups  (see  Brown  and 
Daniel  1990).  Therefore,  we  focus  on  reliability  statis- 
tics only.  See  Tinsley  and  Weiss  (1975)  for  more  on 
agreement  statistics. 

Choice  of  the  proper  reliability  coefficient  depends  on 
whether  it  relates  to  the  ratings  of  individual  observers 
or  to  the  mean  ratings  of  a  group  of  observers.  The  coeffi- 
cient for  individual  observers'  ratings  will  be  called  the 
"observer-to-observer"  coefficient,  while  that  of  observer 
group  mean  ratings  will  be  called  the  "group-to-group" 
coefficient. 

Reliability  coefficients  are  conveniently  computed 
from  a  two-way  ANOVA  of  the  ratings,  which  partitions 
the  sums  of  squares  into  that  for  stimuli,  observers,  and 
observers-by-stimuli  interaction.  The  ANOVA-based 
equation  for  the  observer-to-observer  coefficient  (R0bs) 
is: 


R 


obs 


MSS  -MSe 
MSS+  (n-l)MS, 


[5] 


where 


MSS  =  mean  square  for  stimuli 

MSe  =  mean    square    for  observers-by-stimuli 
interaction 

n     =  number  of  observers  providing  ratings. 

Thus,  the  mean  square  for  observers  does  not  enter  the 
computation  of  observer-to-observer  reliability. 

The  observer-to-observer  reliability  coefficient  is  the 
expected  correlation  between  any  two  observers  drawn 
from  the  relevant  population.  It  indicates  the  reliability 
of  a  single  observer's  ratings  and  is  of  interest  if  deci- 
sions are  to  be  based  on  one  observer's  ratings  of  each 
stimulus,  such  as  typically  occurs  in  an  "expert 
appraisal"  (Daniel  and  Vining  1983). 


Group-to-group  reliability  (Rgrp)  can  be  computed 
from  the  observer-to-observer  coefficient  using  the 
Spearman-Brown  formula  (Nunnally  1978:  211),  or  from 
a  two-way  ANOVA  using  the  following  equation: 


R 


grp 


MSS  -MSe 
MSC 


[6] 


where  MSS  and  MSe  have  the  same  meaning  as  in  [5]. 

Group-to-group  reliability  indicates  the  expected 
correlation  between  group  mean  ratings  for  two  groups 
of  observers  of  the  same  size  as  the  one  at  issue  sampled 
from  the  same  population. 

Reliability  statistics  can  also  be  computed  for  stand- 
ardized metrics,  such  as  Z-scores.  Here,  the  sum-of- 
squares  for  observers  will  always  be  0,  and  the  resulting 
coefficients  are  appropriate  whether  or  not  the  scores  of 
different  observers  (for  R0bs)  or  different  observer  group 
means  (for  Rgrp)  will  be  compared  in  decisionmaking, 
as  differences  in  origin  and  interval  have  already  been 
adjusted. 


Observer-to-Group  Correlation 

The  observer-to-group  correlation  reflects  the  relation- 
ship between  the  ratings  of  an  individual  observer  and 
the  average  ratings  assigned  to  the  same  stimuli  by  the 
rest  of  the  group  of  observers.  The  Pearson  correlation 
of  each  observer's  ratings  with  the  mean  ratings  of  the 
rest  of  the  observers  indicates  the  extent  to  which  any 
observer  is  consistent/inconsistent  with  the  group.  When 
a  high  negative  correlation  between  an  observer  and  the 
group  is  obtained,  it  usually  indicates  that  the  observer 
has  misinterpreted  the  instructions  and  reversed  the  rat- 
ing scale.  Of  course,  a  high  negative  correlation  could 
also  result  if  an  observer  differed  systematically  from  the 
other  observers  in  his  or  her  preferences,  i.e.,  if  the 
observer's  preferences  were  opposite  those  of  the  group. 
When  a  correlation  close  to  0  is  obtained,  it  may  indicate 
that  the  observer  used  an  entirely  different  basis  for  rat- 
ing the  stimuli  than  did  the  rest  of  the  group.  Another 
possibility,  however,  is  that  the  observer  simply  responded 
randomly  or  used  the  wrong  spaces  on  the  response  form. 


Principal  Component  Analysis 

Different  observers  almost  never  agree  perfectly  in 
their  ratings  of  a  set  of  stimuli.  Differences  among 
observers'  ratings  may  be  attributed  to  two  sources: 
(1)  random  variations  in  perception,  preference,  or  use 
of  the  rating  scale;  and  (2)  systematic  differences  in 
preference  or  use  of  the  rating  scale.  When  differences 
among  individual  observers  are  due  only  to  random, 
momentary  variation  in  their  responses,  it  is  appropri- 
ate to  aggregate  their  ratings  into  a  single  scale  using  one 
of  the  methods  provided  by  RMRATE.  In  the  process  of 
combining  different  observers'  ratings,  random  variations 
tend  to  cancel  out,  leaving  a  more  "pure"  measure  of  the 
perceptual  quality  of  the  stimuli.  On  the  other  hand, 
if  differences  among  observers  are  due  to  consistent 
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differences  in  their  underlying  perceptions  of  the 
stimuli,  it  may  be  misleading  to  combine  their  responses 
into  a  single  scale.  If  different  observers  have  differing 
perceptions  of  the  same  stimuli,  it  may  be  more 
appropriate  to  calculate  separate  aggregate  scale  values 
for  different  segments  of  the  observer  population. 

Principal  component  analysis  of  the  inter-rater  corre- 
lation matrix  can  be  used  to  assess  the  homogeneity  of 
a  group's  ratings,  and  to  reveal  the  underlying  dimen- 
sions that  characterize  variations  in  observers'  responses 
to  the  stimuli  (Schroeder  1983,  1987).  This  method 
explains  the  intercorrelations  among  a  set  of  observers 
in  terms  of  a  smaller  set  of  inferred  variables  called 
"components."  Each  component  is  a  weighted  sum  of 
observers'  ratings,  with  the  weights  mathematically 
defined  to  capture  the  maximum  amount  of  observers' 
variance,  subject  to  the  constraint  that  the  components 
must  be  uncorrelated  among  themselves. 

The  use  of  principal  component  analysis  to  uncover 
dimensions  of  preference  is  illustrated  by  Schroeder 
(1987).  Variations  in  preference  for  urban  park  environ- 
ments were  characterized  by  two  principal  components, 
reflecting  the  vegetation  density  and  the  development 
intensity  present  in  the  scenes.  Different  observers 
attached  different  weights  to  each  of  these  two  environ- 
mental dimensions  in  making  their  preference  ratings. 

Principal  component  analysis  produces  three  kinds  of 
output:  (1)  Eigenvalues  represent  the  amount  of  variance 
in  the  total  data  set  that  can  be  accounted  for  by  each 
of  the  components.  A  principal  component  analysis  is 
usually  done  on  standardized  scores,  so  that  each 
observer's  variance  is  equal  to  1.  The  total  variance  in 
the  data  set  (i.e.,  the  sum  of  the  individual  observer's 
variances)  is  therefore  equal  to  the  number  of  observers. 
The  proportion  of  the  total  variance  accounted  for  by  a 
component  is  calculated  by  dividing  the  eigenvalue  for 
that  component  by  the  total  number  of  observers  in  the 
group.  (2)  Component  loadings  measure  how  strongly 
each  observer's  ratings  are  related  to  each  of  the  com- 
ponents. If  an  observer  has  a  high  loading  on  a  particu- 
lar component,  it  means  that  the  component  is  highly 
correlated  with  the  observer's  ratings.  (3)  Component 
scores  show  the  location  of  each  of  the  stimuli  on  each 
of  the  dimensions  represented  by  the  components. 

The  first  principal  component  represents  the  best  fit 
to  the  group's  ratings  that  can  be  obtained  with  a  one- 
dimensional  scale.  The  proportion  of  variance  accounted 
for  by  the  first  component  is  a  measure  of  the  consensus 
among  the  observers.  For  example,  if  the  first  compo- 
nent accounts  for  60%  of  the  variance  in  the  data  set, 
it  means  that  more  than  half  of  the  variance  in  the  rat- 
ings can  be  captured  in  a  one-dimensional  scale.  This 
scale  represents  the  majority  "consensus"  about  the  per- 
ceptual quality  of  the  stimuli.  The  variance  associated 
with  the  remaining  components  represents  variations 
from  this  consensus.  The  proportion  of  variance  ac- 
counted for  by  the  second  component  indicates  how 
much  additional  variance  can  be  explained  when  a 
second  dimension  is  introduced.  The  first  two  compo- 
nents taken  together  represent  the  best  fit  to  the  data  that 
can  be  achieved  with  two  perceptual  dimensions. 


Similar  interpretations  hold  for  the  third  and  subsequent 
components. 

In  many  applications  of  principal  component  analysis, 
only  components  with  eigenvalues  greater  than  1  are 
considered  to  be  significant,  i.e.,  to  represent  nonran- 
dom  sources  of  variance  in  the  data.  This  standard 
criterion  is  rather  arbitrary,  however.  To  determine 
whether  components  beyond  the  first  represent  signifi- 
cant variations  from  the  majority  viewpoint,  at  least  three 
criteria  should  be  considered:  (1)  the  size  of  the  compo- 
nents, (2)  the  number  of  observers  having  their  strong- 
est loadings  on  each  component,  and  (3)  the  interpreta- 
bility  of  the  components.  If  a  component  accounts  for 
substantial  variance  (e.g.,  10%  or  more),  if  several 
observers  have  their  strongest  loadings  on  that  compo- 
nent (e.g.,  3  or  more),  and  if  stimuli  with  high  compo- 
nent scores  and  stimuli  with  low  component  scores  differ 
with  respect  to  some  identifiable  physical  attribute  (e.g., 
tree  density),  then  that  component  probably  represents 
real  (nonrandom)  variation  in  the  observers'  perceptions. 

Other  approaches  besides  principal  component  anal- 
ysis can  also  be  used  to  examine  individual  differences 
among  raters,  for  example,  cluster  analysis  and  multi- 
dimensional scaling.  Cluster  analysis  differs  from  the 
approach  presented  here  because  it  assumes  that  raters 
fall  into  distinct  groups  and  that  preferences  are  rela- 
tively homogeneous  within  the  groups.  Preference  var- 
iations, however,  might  not  take  this  form.  For  exam- 
ple, preferences  could  vary  continuously  along  one  or 
more  dimensions  without  there  being  any  clear  clusters. 
It  is  beyond  the  scope  of  this  report  to  provide  a  com- 
plete selection  of  techniques  for  analyzing  individual 
differences.  If  the  principal  component  analysis  suggests 
that  there  are  meaningful  variations  in  preference,  the 
user  may  gain  additional  insight  by  analyzing  the  data 
using  cluster  analysis  or  other  techniques  that  are  avail- 
able in  standard  statistical  packages. 

If  principal  component  analysis  is  used,  it  is  best  not 
to  delete  observers  who  have  low  correlations  with  the 
group  average.  Such  deletion  increases  the  homogeneity 
of  the  observer  group  by  removing  observers  who  devi- 
ate from  the  majority  consensus,  but  it  is  precisely  these 
observers  that  the  principal  component  analysis  is 
designed  to  detect.  Therefore,  deletion  of  observers 
defeats  the  purpose  of  the  principal  component  analysis 
option,  and  the  two  should  not  be  used  together. 


USER'S  GUIDE  TO  RMRATE 

RMRATE  requires  560K  bytes  of  random  access 
memory  (RAM)  to  execute,  as  would  typically  be  avail- 
able on  a  computer  with  640K  bytes  of  RAM.  Upon 
special  request  to  the  authors,  a  copy  that  will  run  in 
less  memory  can  be  made  available,  but  it  should  be 
noted  that  sample  sizes  that  can  be  analyzed  with  such 
a  version  are  also  smaller.2 

2RMRA  TE  has  also  been  successfully  compiled  and  run  under  OS/2, 
removing  the  640K  barrier  imposed  by  DOS  and  making  expanded 
memory  available  for  larger  sample  sizes. 
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Regardless  of  the  RAM  requirements,  two  versions  of 
RMRATE  are  available.  One  utilizes  a  math  coprocessor 
and  will  only  run  on  a  computer  with  a  coprocessor.  The 
other  does  not  utilize  a  coprocessor.  It  will  run  on  a  com- 
puter that  has  a  coprocessor,  but  will  of  course  run  more 
slowly  than  the  coprocessor  version.  Both  versions  are 
supplied  to  the  user. 

RMRATE  accommodates  the  maximum  number  of  ob- 
servers and  stimuli  that  memory  allows.  If  the  data  set 
is  too  large,  a  table  is  printed  of  possible  combinations 
of  observers  and  stimuli  that  can  be  accommodated  and 
the  run  is  terminated. 

The  principal  component  analysis  requires  more 
memory  than  the  other  analyses.  If  memory  is  sufficient 
to  accommodate  the  data  set  for  all  analyses  except  the 
principal  component  analysis,  the  user  can  select  a  sub- 
set of  the  observers  to  be  included  in  the  principal  com- 
ponent analysis,  or  a  subset  of  the  observers  is  randomly 
selected  for  this  analysis  and  the  number  of  observers 
included  in  the  analysis  is  noted. 

RMRATE  requires  two  sets  of  input:  data  (the  ratings 
to  be  analyzed)  and  program  control  information.  The 
data  should  be  in  a  separate  file,  such  as  that  depicted 
in  appendix  A.  The  control  information  can  be  entered 
directly  from  the  keyboard,  but  a  preferred  way  is  to  pre- 
pare a  control  file  using  an  editor,  and  enter  that  file  to 
begin  the  run.  Appendix  B  gives  an  example  of  a  con- 
trol file  for  analyzing  the  data  in  appendix  A.  On  a  PC, 
the  control  file  could  be  entered  into  RMRATE  using  the 
DOS  redirection  feature  (<)  on  the  command  line.  For 
example,  if  the  control  file  were  called  SAMPLE. RUN, 
the  program  would  be  started  by  entering  RMRATE 

<  SAMPLE. RUN. 

RMRATE  output  is  organized  into  19  displays,  which 
are  individually  selected  by  the  user.  On  a  PC,  the 
requested  output  will  simply  be  presented  on  the  moni- 
tor unless  the  user  intervenes.  Of  course,  output  to  the 
monitor  can  be  simultaneously  printed  using  the  DOS 
"Control  Print  Screen."  In  most  cases,  however,  the  best 
option  is  to  redirect  the  output  to  a  file,  which  can  then 
be  accessed  for  viewing  or  printing.  On  a  PC,  the  out- 
put can  be  redirected  to  a  file  by  using  the  DOS  redirec- 
tion feature  ( > )  on  the  command  line.  To  continue  with 
the  above  example,  if  the  output  file  were  called 
SAMPLE. OUT,  the  command  line  would  be  RMRATE 

<  SAMPLE. RUN  >  SAMPLE. OUT.  Appendixes  C  and 
D  contain  the  RMRATE  output  that  the  data  and  input 
parameters  in  appendixes  A  and  B  would  produce. 

Input 

Seventeen  lines  (or  sets  of  lines)  of  program  control  in- 
formation are  required.  Each  line  is  described  below.  The 
reader  can  compare  the  control  information  in  appen- 
dix B  with  the  data  in  appendix  A  to  see  an  example  of 
how  a  control  file  might  be  arranged.  Detailed  instruc- 
tions for  preparing  a  control  file,  including  column 
specifications,  are  given  in  the  "Input  Parameters"  list- 
ing at  the  end  of  this  section.  Unless  noted  otherwise, 
the  examples  in  the  "Input  Parameters"  listing  are 


identical  to  the  sample  runstream  of  appendix  B. 

Line  lengths  are  limited  to  80  characters.  Most  of  the 
information  required  on  each  of  the  17  lines  can  be  eas- 
ily contained  within  80  characters;  some,  however,  may 
require  more  than  80  characters  and,  thus,  will  require 
more  than  one  line.  Numeric  input  should  be  right- 
justified  within  the  specified  column  widths;  alphanu- 
meric input  should  be  left-justified. 

Line  1.  Title. — Enter  a  title;  it  will  appear  on  each  page 
of  output,  and  can  be  used  to  identify  the  run,  data  set, 
and  other  pertinent  information.  Date  and  time  are  also 
provided  by  the  program,  to  distinguish  among  runs  with 
the  same  title. 

Line  2.  Size  of  data  set. — Enter  the  numbers  of  observ- 
ers and  stimuli.  RMRATE  will  calculate  the  amount  of 
computer  memory  required  to  process  the  user's  data  set. 
If  the  data  set  is  too  large  for  the  principal  component 
analysis  portion  of  RMRATE,  but  not  too  large  for  other 
portions  of  the  program,  the  output  will  reveal  that  the 
maximum  number  of  observers  to  be  included  in  the 
principal  component  analysis  is  less  than  the  total  num- 
ber of  observers.  Further,  if  the  data  set  is  even  too  large 
for  other  analyses,  an  error  message  will  be  printed  along 
with  a  table  of  possible  combinations  of  reduced  num- 
bers of  observers/stimuli  that  will  fit  in  the  program's 
memory,  and  the  run  will  be  terminated. 

Line  3.  Input  option. — The  data  may  be  entered  by  ob- 
server or  by  stimulus.  Leaving  this  line  blank  indicates 
that  the  case  for  analysis  is  an  observer — that  an  observ- 
er's ratings  for  all  stimuli  are  read  in  before  proceeding 
to  another  observer.  Entering  a  1  indicates  that  the  case 
for  analysis  is  a  stimulus — that  all  observers'  ratings  for 
a  single  stimulus  are  read  in  before  proceeding  to  another 
stimulus.  Typically  ratings  are  entered  by  observer.  This 
is  likely  to  be  the  case,  for  example,  when  ratings  were 
taken  from  forms  that  individual  observers  have  com- 
pleted. Ratings  might  be  organized  by  stimulus,  for  exam- 
ple, when  all  observers  simultaneously  recorded  their 
ratings  on  keyboards  to  a  central  receiving  station  that 
stores  all  responses. 

Line  4.  Name  of  data  file. — This  name  may  contain  up 
to  8  characters,  with  an  extension  of  up  to  3  characters, 
following  standard  DOS  for  personal  computers.  A  full 
path  may  be  included,  if  needed. 

Line  5.  Input  format  of  data  — A  standard  FORTRAN 
format  statement,  of  type  integer,  is  required  describing 
the  format  of  each  case  of  data.  A  FORTRAN  statement 
allows  two  ways  of  skipping  over  unwanted  information 
or  blank  spaces  in  a  file:  (1)  A  "T"  can  be  used  to  move 
to  an  indicated  column;  for  example,  a  "T4"  indicates 
that  what  follows  will  begin  in  column  4.  (2)  An  "X" 
can  be  used  to  skip  a  given  number  of  columns;  for  exam- 
ple, a  "3X"  indicates  that  the  next  three  columns  are  to 
be  skipped.  Data  to  be  read  are  indicated  with  an  "I", 
with  the  number  of  columns  per  rating  value  listed  after 
the  I  and  the  number  of  values  in  succession  listed  before 
the  I;  for  example,  a  "512"  indicates  that  5  values  will 
be  read  in  two-column  fields.  Only  the  integer  (I)  format 
is  allowed.  Decimal  points  are  not  allowed  within  the 
columns  where  data  are  to  be  read.  Finally,  if  the  data 
for  each  case  are  to  be  read  from  more  than  one  line,  a 
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"/"  is  used  to  indicate  the  end  of  a  line.  The  format  state- 
ment in  appendix  B  (T22,  1312)  indicates  that  the  13  two- 
column  ratings  of  each  case  (which  is  a  single  stimulus, 
as  indicated  by  line  3  of  appendix  B)  begin  in  column 
22.  The  first  21  columns  of  the  sample  data  file,  which 
contain  additional  information  about  the  stimuli,  are 
ignored. 

Line  6.  Number  of  rating  values. — Enter  the  number 
of  possible  responses  on  the  rating  scale  used  by  the 
observers.  RMRATE  requires  this  to  be  a  number 
between  5  and  10  (i.e.,  the  largest  rating  scale  permit- 
ted is  a  10-point  scale). 

Line  7.  Output  rating  scale. — First,  enter  the  value  to 
be  printed  in  the  output  to  indicate  that  the  stimulus  was 
not  rated  by  an  observer  (that  the  rating  is  missing).  Only 
numerical  values,  in  a  field  of  up  to  four  characters,  are 
permitted.  Thus,  possibilities  for  indicating  a  missing 
value  include  a  single-digit  number  (e.g.,  0),  a  negative 
number  (e.g.,  -9),  and  a  two-digit  number  (e.g.,  99).  If 
this  field  is  left  blank,  RMRATE  will  print  a  "0"  to  indi- 
cate a  missing  value. 

Then  enter  each  value  of  the  rating  scale  to  be  used 
in  the  output,  beginning  with  "least  preferred"  (lowest 
perceived  value)  and  ending  with  "most  preferred" 
(highest  perceived  value).  This  scale  would  normally  be 
identical  to  the  rating  scale  used  by  the  observers.  Line 
7  of  appendix  B  indicates  that  a  0  is  to  be  used  in  out- 
put to  indicate  no  response,  and  that  a  10-point  scale 
is  to  be  used  with  a  "1"  used  to  indicate  "least 
preferred,"  etc. 

Line  8.  Input  rating  scale. — First,  enter  the  value  used 
in  the  data  to  indicate  that  a  stimulus  was  not  rated  by 
an  observer  (that  the  rating  is  missing).  Note  that 
RMRATE  does  not  distinguish  between  zeros  and 
blanks,  so  that  if  "0"  is  one  of  the  rating  values,  a  blank 
could  not  be  used  to  indicate  a  missing  rating  value. 
Then  enter  the  values  of  the  rating  scale  in  which  the 
observers'  responses  are  recorded  in  the  data  file,  again 
from  "least  preferred"  to  "most  preferred."  Line  8  of 
appendix  B  indicates  that  a  "-1"  is  used  in  the  data  file 
(appendix  A)  to  indicate  no  response  and  that  the  ratings 
are  in  terms  of  a  0-9  scale.  A  0-9  scale  is  sometimes  used 
to  compress  10-point  scale  responses  to  one  column 
each,  but  of  course  this  procedure  can  only  be  used  with 
RMRATE  if  there  are  no  blanks  since  a  0  is  synonymous 
with  a  blank.  Because  a  "-1"  was  used  to  indicate  miss- 
ing data,  the  data  in  appendix  A  could  not  be  com- 
pressed to  one  column  per  rating  value. 

Line  9.  Number  of  conditions. — Enter  the  number  of 
conditions,  including  the  baseline,  if  any.  A  "condition" 
is  simply  a  group  of  stimuli.  For  example,  if  the  stimuli 
are  landscape  scenes,  a  condition  might  be  all  the  scenes 
from  a  certain  park  or  forest  area.  A  maximum  of  20  con- 
ditions can  be  specified.  Line  9  of  appendix  B  indicates 
that  3  conditions  will  be  used  in  the  example.  If  all 
stimuli  are  of  the  same  condition  (or  no  condition  dis- 
tinctions are  desired),  enter  a  "1". 

Line  10.  Condition  definitions. — Enter  one  line  for 
each  condition.  Each  line  gives  the  condition  name  and 
then  indicates  which  stimuli  are  included  in  that  con- 
dition. If  there  is  only  one  condition  (a  "1"  was  entered 


on  line  9),  a  name  must  still  be  entered,  and  RMRATE 
considers  all  stimuli  to  be  the  baseline.  The  baseline  is 
important  in  several  of  the  scaling  methods  used  by 
RMRATE  (see  Brown  and  Daniel  1990).  If  there  is  more 
than  one  condition,  the  baseline  condition  must  be  listed 
first;  it  would  normally  be  named  "baseline,"  but  any 
name  can  be  used.  RMRATE  does  not  allow  specifica- 
tion of  more  than  one  condition  without  one  of  them 
being  the  baseline.  Thus,  one  could  not  specify  two  con- 
ditions and  use  all  stimuli  as  the  baseline.3 

The  stimuli  included  in  a  condition  are  indicated  by 
sequence  number  (the  order  in  which  they  occur  on  the 
data  file).  There  are  three  options  for  indicating  stimu- 
lus sequence  numbers:  (1)  The  list  option  allows  the 
sequence  numbers  of  the  stimuli  to  be  specifically 
enumerated  (not  necessarily  in  sequential  order).  The 
BASELINE  condition  is  specified  in  appendix  B  using 
this  option,  showing  that  10  stimuli  (the  first  through 
the  tenth  stimulus)  are  in  this  condition.  (2)  The  pattern 
option  allows  every  nth  stimulus  to  be  easily  assigned 
to  a  condition.  The  MIXPRE  condition  is  specified  in 
appendix  B  using  this  option  and  has  14  stimuli  (11,  12, 

13  24).  (3)  The  final  option  allows  all  stimuli  not 

yet  assigned  to  a  condition  to  be  assigned  to  the  desig- 
nated condition.  The  MIXPOST  condition  is  specified 
in  appendix  B  using  this  option.  If  only  one  condition 
is  listed,  it  must  still  be  named,  and  the  included  stimuli 
must  be  designated,  which  can  easily  be  done  using  the 
third  option. 

Line  11.  Stimulus  names. — A  name  may  be  given  to 
each  stimulus  for  identification  in  the  output.  List  each 
name  on  a  separate  line.  The  order  must  match  the  order 
in  which  the  stimulus  ratings  are  listed  in  the  data  file. 
If  names  are  not  desired,  enter  "NO  NAMES,"  and  the 
stimuli  will  be  identified  in  the  output  by  their  input 
sequence  numbers. 

Line  12.  Removal  controls. — Observers  or  stimuli  may 
be  removed  from  the  analysis  if  they  do  not  meet  certain 
criteria.  The  following  three  criteria  are  available  for 
removing  observers:  (1)  excessive  missing  ratings,  (2) 
inadequate  range  of  rating  values  used,  and  (3)  inade- 
quate correlation  of  an  observer's  ratings  with  the  mean 
ratings  for  the  group  of  all  other  observers.  A  stimulus 
may  be  removed  if  an  excessive  number  of  observers  did 
not  rate  the  stimulus.  If  missing  ratings  remain  after  the 
removal  specifications  have  been  processed,  each  miss- 
ing rating  is  replaced  with  the  respective  observer's  mean 
rating  before  analysis  begins.  This  replacement  occurs 
even  if  most  of  an  observer's  ratings  are  missing,  as  long 
as  the  observer  meets  the  other  criteria  for  inclusion. 

To  avoid  biasing  the  results,  the  range-of-rating-scale 
and  correlation  criteria  for  removing  observers  must  be 
used  with  great  caution.  The  principal  reason  for  the 
former  criterion  is  to  remove  observers  who  showed  no 
discrimination  among  the  stimuli.  If  an  observer  used 
a  uniform  rating  for  all  stimuli  (e.g.,  assigned  all  "5"s), 

37b  simulate  this,  one  could  duplicate  the  data  set  so  that  the  input 
data  file  contained  two  sequential  sets  of  the  same  data,  and  then  indi- 
cate in  the  control  file  that  the  first  set  was  the  baseline  and  that  the  second 
set  contained  the  two  other  conditions. 
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some  of  the  analyses  would  be  mathematically  unde- 
fined; thus,  the  default  range  is  1.  Users  have  the  option, 
however,  to  remove  observers  who  showed  some,  but 
insufficient,  discrimination  among  the  stimuli.  The 
other  criterion  allows  the  user  to  remove  observers 
whose  ratings  correlate  too  poorly  with  the  mean  ratings 
of  the  other  observers.  This  may  be  used,  for  example, 
to  remove  observers  whose  correlation  with  the  group 
of  all  other  observers  is  strongly  negative  (e.g.,  a  -0.7), 
which  could  indicate  that  the  observer  misinterpreted 
the  rating  scale,  confusing  "most  preferred"  with  "least 
preferred,"  and  vice  versa.  Again,  care  should  be  exer- 
cised when  using  these  criteria. 

Line  13.  Exclusion  of  observers. — This  line  allows  ob- 
servers to  be  unconditionally  excluded  from  the  analy- 
sis. The  input  format  requires  listing  ranges  of  observers 
to  be  excluded,  indicated  by  the  sequence  number  of  the 
first  and  last  observer  in  the  range. 

Line  14.  Exclusion  of  stimuli. — This  line  allows 
stimuli  to  be  unconditionally  excluded  from  the  analy- 
sis. As  in  line  13,  the  input  format  requires  listing  ranges 
of  stimuli  to  be  excluded. 

Line  15.  Principal  component  analysis  controls. — The 
user  has  the  option  of  choosing  the  number  of  principal 
components  for  which  component  loadings  and  compo- 
nent scores  will  be  included  in  output.  This  can  be  done 
by  specifying  either  the  number  of  components  to  be 
included  or  the  minimum  eigenvalue  for  a  component 
to  be  included.  If  a  specific  number  of  components  is 
specified,  RMRATE  chooses  the  components  to  include 
in  order  of  decreasing  eigenvalue.  If  the  user  does  not 
specify  one  of  these  options,  all  components  with  an 
eigenvalue  greater  than  or  equal  to  1.0  will  be  included 
in  the  output. 

The  principal  component  analysis  requires  more 
memory  for  a  given  data  set  than  do  the  other  analyses. 
If  all  observers  cannot  be  included  in  this  analysis,  users 
have  two  options  for  reducing  the  number  of  observers. 
The  first  option  is  to  let  RMRATE  randomly  select  the 
observers  to  be  included.  RMRATE  will  select  the  max- 
imum number  possible.  The  random  number  generator 
in  RMRATE  (L'Ecuyer  1988)  can  take  its  seed  from  the 
date  and  time  on  the  computer's  clock;  this  is  the  default 
option.  However,  the  user  can  enter  seeds  (two  are 
required),  which  allows  the  user  to  replicate  the  selec- 
tion of  observers  (demonstrating  that  the  "random  num- 
ber" generator  is  not  truly  random,  but  is  the  best 
approximation  available).  The  second  option  is  to 
specify  which  observers  are  to  be  included  in  the 


principal  component  analysis.  This  can  be  done  by 
specifying  a  pattern  (as  described  for  input  line  10)  or 
by  specifying  the  sequence  number  of  each  observer  to 
be  included. 

The  principal  component  analysis  can  be  quite  time 
consuming.  The  time  required  to  perform  the  analysis 
increases  geometrically  as  the  number  of  observers 
increases.  On  a  PC,  the  time  required  for  large  data  sets 
may  seem  unduly  long,  and  since  most  of  the  matrix 
operations  are  performed  in  memory,  there  is  no  visible 
evidence  that  RMRATE  is  not  hung.  Have  patience.  On 
a  timesharing  mainframe,  the  costs  may  increase  dramat- 
ically for  large  data  sets. 

Line  16.  Output  options. — The  available  output  con- 
sists of  19  displays.  The  displays  consist  of  one  or  more 
tables  or  two-dimensional  plots.  Each  of  the  displays  is 
described  in  the  "Output"  section,  below.  Example  dis- 
plays are  presented  in  appendix  C. 

In  addition  to  being  printed  or  written  to  a  file,  some 
of  the  displays  can  also  be  written  to  a  file  in  more 
machine-readable  form  for  use  as  input  in  other  pro- 
grams. This  option  is  available  for  displays  1,  2,  4,  and 
9  through  12.  Appendix  D  gives  an  example  of  the  more 
machine-readable  version  of  display  4. 

The  user  indicates  which  displays  are  desired  in  the 
first  19  columns  of  line  16.  A  "1"  is  entered  in  the 
column  if  the  user  wants  the  display  represented  by  that 
column  to  be  included  as  output,  and  a  "2"  is  entered 
in  the  column  if  the  display  is  wanted  both  in  the  regu- 
lar output  and  in  more  machine-readable  form.  Column 
1  represents  display  1,  column  2  represents  display  2, 
etc.  The  user  can  use  the  column  indicator  of  an  editor 
program  to  keep  track  of  the  columns. 

At  the  right-hand  end  of  line  16,  the  absolute  value 
of  the  correlation  coefficient  cutoff  for  printing  plots  of 
pairs  of  scaled  values  is  entered.  Scattergrams  are  gener- 
ated for  those  pairs  of  statistics  in  display  6  whose  corre- 
lation coefficient  (absolute  value)  is  less  than  the 
threshold.  See  the  discussion  of  display  8  in  the  "Out- 
put" section  for  more  on  this  option. 

Line  17.  "Saved  output"  file  name. — This  line  gives 
the  name  of  the  file  to  which  tables  are  written  in  more 
machine-readable  form.  All  available  displays  requested 
with  a  "2"  on  line  16  are  written  to  this  one  file.  A  file 
by  this  name  may  not  already  exist  on  the  directory  to 
which  the  file  is  directed.  This  line  must  be  included, 
even  if  no  output  is  to  be  saved  in  more  machine- 
readable  form.  If  no  machine-readable  tables  are  wanted, 
enter  "NONE." 
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INPUT  PARAMETERS 


Line  Parameters  and  descriptions 


Columns  Example 


Comments  on  example 


1  Title.  80  characters  or  less  are  entered  to  identify  the  data  set,  the 
run,  or  other  pertinent  information. 

2  Memory  requirements.  Enter  the  number  of  observers  and  the 
number  of  stimuli. 


1-80 


1-5 
6-10 


SAMPLE  RUNSTREAM 


13 
35 


13  observers, 
35  stimuli 


3  Input  option. 

Leave  blank  if  input  is  by  observers:  an  observer's  ratings  for  all 
stimuli  are  read  in  before  proceding  to  another  observer. 

Enter  1  if  input  is  by  stimuli:  all  observers'  ratings  for  a  single 
stimulus  are  read  in  before  proceding  to  another  stimulus. 

4  Ratings  (input  data)  file  name. 

5  Input  format.  Standard  FORTRAN  format  statement  describing  the 

input  ratings  on  the  file  named  above.  Must  be  INTEGER  type, 
enclosed  by  parentheses. 


1-80 
1-80 


SAMPLE.DAT 
(T22.13I2) 


Input  by  stimuli 


PC  style  file  name 

Example  1:  Reads  13 
values  per  line  in  2- 
column  fields  beginning 
in  column  22 


1-80 


(10X,15I2/5X,20I2) 


6  Number  of  rating  values,  not  including  the  value  used  to  indicate 
missing  data  (5  <  N  <  10).  This  is  the  number  of  rating  values 
permitted  to  an  observer. 

7  Rating  scale  desired  in  output. 

Enter  the  "not  rated"  (i.e.,  missing)  value. 


In  subsequent  5  column  fields,  begin  with  the  "least  preferred" 
value  and  proceed  to  the  "most  preferred"  value.  The  number 
of  values  should  correspond  with  the  information  on  input 
line  6. 


Rating  scale  used  for  input.  Enter  the  "not  rated"  (i.e.,  missing) 
value.  In  subsequent  5  column  fields,  beginning  with  the  "least 
preferred"  value,  enter  the  scale  found  in  the  data  file  named  on 
input  line  4.  This  scale  may  be  identical  to  that  on  line  7,  but  it 
must  still  be  repeated  here. 


9    Number  of  conditions,  including  the  baseline  (1  <  N  <  20). 


4-5 


1-5 


6-10 

11-15 

16-20 

21-25 

26-30 

31-35 

36-40 

41-45 

46-50 

51-55 

1-5 


6-10 

11-15 

16-20 

21-25 

26-30 

31-35 

36-40 

41-45 

46-50 

51-55 

4-5 


10 


1 
2 
3 
4 
5 
6 
7 
8 
9 
10 


Example  2  (not  included 
in  appendix  B):  Reads 
15  values  on  first  line 
in  2-column  fields,  begin- 
ning in  column  1 1 ,  then 
20  values  on  second  line, 
beginning  in  column  6 

10  possible  values 
in  the  rating  scale 
besides  blank 


A  blank  =  missing; 
RMRATE  will  print  a  0 

RMRATE  will  print 
results  in  terms  of  a 
10-point  scale 


A  -1  indicates  a 
missing  value 

The  data  file  lists 
ratings  on  a  0-9 
scale 


Baseline  and  2 
conditions 
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Line  Parameters  and  descriptions 


Columns  Example 


Comments  on  example 


10    Condition  definitions.  There  must  be  one  line  for  each  condition 
specified  on  line  9.  The  baseline  condition  must  be  first. 

Condition  name. 


Stimuli  belonging  to  this  condition 
To  specifically  enumerate  the  stimuli  belonging  to  a  condition, 
enter  the  number  of  stimuli  belonging  to  the  condition; 
enter  in  subsequent  5-column  fields  the  sequence  number 
of  each  stimulus  belonging  to  this  condition.  Continue  on 
subsequent  lines,  if  necessary.  Stimuli  need  not  be  listed  in 
numerical  order. 


1-10 


11-15 
16-20 
21-25 
26-30 
31-35 
36-40 
41-45 
46-50 
51-55 
56-60 
61-65 


BASELINE 


10 
1 
2 
3 
4 
5 
6 
7 
8 
9 

10 


The  name  of  the 
first  condition 


Stimuli  1-10  belong 
to  this  condition 


Condition  name.  1-10 

Stimuli  belonging  to  this  condition. 
To  specify  a  pattern  for  stimuli  belonging  to  this  condition,  enter 

a  0;  11-15 

enter  the  sequence  number  of  the  1st  stimulus  in  the  pattern;  16-20 

enter  the  increment  in  the  pattern;  21-25 

enter  the  sequence  number  of  the  last  stimulus  in  the  pattern.  26-30 


MIXPRE 


0 
11 

1 

24 


The  name  of  the 
second  condition 


Stimuli  11, 
12,  .  .  . ,  24  are 
all  assigned  to  this 
condition 


Condition  name. 


1-10 


MIXPOST 


The  name  of  the  third 
condition 
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Stimuli  belonging  to  this  condition. 
Enter  -1  to  cause  all  stimuli  not  yet  assigned  to  another  11-15  -1 

condition  to  this  condition.  This  specification  can  only  be  used 
with  the  last  (or  only)  condition. 

Stimuli  names.  If  no  names  are  assigned  to  the  stimuli,  enter  1-10  NO  NAMES 

NO  NAMES.  If  each  stimulus  has  a  name  or  identifier,  use  one 
line  for  each  stimulus  (one  name  per  line),  with  the  name  in 
columns  1-10. 


All  stimuli  not  yet 
assigned  to  an  earlier 
condition 

RMRATE  will  assign 
each  stimulus'  input 
order  no.  as  its 
"name" 


12    Removal  controls. 


Excessive  missing  ratings  by  an  observer.  Enter  the  number  of  1-5 
missing  responses  which  would  be  unacceptable  and  cause  the 
observer  to  be  removed  from  the  analyses  (1  <  N  <  number  of 
stimuli  +  1).  If  none  are  to  be  removed  regardless  of  the  number 
of  missing  ratings,  enter  number  of  stimuli  +  1. 

Excessive  missing  ratings  among  all  observers  for  a  single  6-10 
stimulus  (1  <  N  <  number  of  observers  +  1).  If  none  are  to  be 
removed  regardless  of  the  number  of  missing  ratings,  enter 
number  of  observers  +  1. 


Any  observer  leaving 
1  or  more  ratings  blank 
will  be  removed 


Any  stimulus  which 
was  not  rated  by  1  or 
more  observers  will 
be  removed 


Poor  correlation  of  an  observer's  ratings  with  rest  of  the  group's  11-15 
ratings.  Enter  threshold  r  coefficient,  below  which  an  observer  will 
be  removed.  If  none  are  to  be  removed  regardless  of  correlation 
with  the  group,  enter  -2.0. 

Poor  range  of  ratings.  If  an  observer  used  a  uniform  rating  value  16-20 
for  all  stimuli,  some  of  the  analyses  would  break  down  math- 
ematically. Thus,  the  default  minimum  range  for  inclusion  is  a  1, 
but  the  user  may  require  a  greater  range  if  desired.  If  left  blank, 
the  default  is  used. 


-.7 


Any  observer  with 
r  <  -.7 

will  be  removed 


Blank  =  default 
(range  of  1) 
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Line  Parameters  and  descriptions 


Columns  Example 


Comments  on  example 


13  Exclusion  of  observers.  Leave  line  blank  if  no  observers  are  to  be 

unconditionally  excluded.  Enter  pairs  of  numbers  in  5-column 
fields  as  shown  to  the  right  to  identify  ranges  of  observers  to  be 
unconditionally  excluded.  The  numbers  correspond  to  input  order; 
i.e.,  entering  a  10  would  cause  the  tenth  observer  to  be  excluded. 
Each  pair  represents  a  series  (a  range)  of  observers  to  be 
excluded:  the  first  number  represents  the  first  observer  in  a  series 
and  the  second  represents  the  last  observer  of  the  series.  If  a 
series  consists  of  only  one  observer,  enter  that  number  in  the 
first  position  AND  in  the  second  position,  or  the  second  position 
may  be  left  blank.  Continue  entering  pairs  as  required,  and  con- 
tinue on  subsequent  lines  if  necessary.  Terminate  the  list  with  at 
least  one  pair  of  blank  columns,  even  if  another  line  is  required. 

14  Exclusion  of  stimuli.  Same  instructions  as  for  exclusion  of  observers 

in  line  13  above,  except  the  numbers  represent  the  input  order  of 
the  stimuli.  Again,  a  blank  line  means  the  user  is  not  uncondi- 
tionally excluding  any  stimulus. 


1-5 
6-10 


11-15 
16-20 


I-  5 
6-10 

II-  15 
16-20 


13 


17 


Exclude  the  third 
observer 


Blank  pair  to  end 
list  of  exclusions 


Exclude  thirteenth 
observer 

Exclude  seventeenth 
observer 


21-25 
26-30 


32 
33 


Excluded  observers 
32  to  33 


31-35 
36-40 


Blank  pair  to  end 
list  of  exclusions 


15    Principal  component  analysis  controls.  If  column  13  of  input  line  16 
does  not  indicate  that  the  principal  component  analysis  is  to  be 
done,  leave  this  line  blank.  If  the  principal  component  analysis  is  to 
be  done,  but  the  defaults  are  acceptable,  it  may  also  be  left  blank. 

Number  of  components  to  be  printed. 


Minimum  eigenvalue  for  a  component  to  be  printed.  Default  =  1.0. 
Note  that  if  both  the  number  of  components  to  be  printed  and  the 
minimum  eigenvalue  are  specified,  the  one  which  will  print  the 
fewest  components  determines  the  number  to  be  displayed. 

Identifying  observers  to  be  included  in  the  principal  component 
analysis. 

Automatic  identification. 

Enter  a  0  or  blank.  RMRATE  will  include  all  observers,  if 
possible. 

Enter  a  numerical  seed  of  1  to  5  digits  for  the  random  number 
generator  (or  leave  blank  and  RMRATE  will  pick  a  seed) 
which  will  randomly  select  observers  to  include  when  it  is 
not  possible  to  use  all  of  them. 

Enter  a  second  seed  for  the  random  number  generator  (or 
leave  blank— if  a  seed  was  entered  in  the  previous  field, 
one  must  also  be  entered  here). 

To  specify  a  pattern, 
enter  -1; 

enter  the  sequence  number  (input  order)  of  the  1st  observer 

in  the  pattern; 
enter  the  increment  in  the  pattern; 

enter  the  sequence  number  of  the  last  observer  in  the  pattern. 


To  specifically  enumerate  the  observers  to  be  included,  enter 
in  5-column  fields  the  sequence  number  of  each  observer 
to  be  included.  Continue  on  additional  lines,  if  necessary. 


1-5 


6-10 


11-15 


16-20 


21-25 


11-15 
16-20 

21-25 
26-30 


11-15 
16-20 
21-25 
26-30 
31-35 
36-40 
41-45 
46-50 
51-55 


-1 
3 

4 
247 


17 
33 
65 
83 
90 
91 
92 
105 
107 


Blank  =  default 

(to  be  determined  on 

basis  of  eigenvalues) 

Blank  =  default 
(print  all  components 
whose  eigenvalues 
are  >  1.0) 


Example  1: 
Blank  =  0 

Blank  =  RMRATE  picks 
seed 


Blank  =  RMRATE  picks 
seed 


Example  2  (not  in 
appendix  B): 


Observers  3,  7,  11, 
. . . ,  247  are  all  to  be 

included  in  the  factor 

analysis 

Example  3  (not  in 
Appendix  B): 
Include  9  obs.:  17, 
33,  65,  83,  90,  91, 
92,  105,  107 
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Line  Parameters  and  descriptions 


Columns  Example 


Comments  on  example 


16    Output  options. 


Output  display  selection.  Each  column  represents  the  display  by 
the  same  number.  If  the  display  is  to  be  omitted,  leave  the 
column  blank. 

If  the  display  is  to  be  generated,  enter  a  1.  If  the  display  can  be 
written  to  the  "saved  output"  file  (see  line  17  below)  in  more 
machine-readable  form,  and  if  this  additional  output  is  desired, 
enter  2  for  the  display  to  be  both  generated  and  saved. 

Display  1.  Raw  ratings  and  associated  statistics.  (May  be  saved.)  1 

Display  2.  Ratings  and  associated  statistics.  (When  this  display  2 
requested,  it  will  be  generated  only  if  observers  and/or 
stimuli  are  removed  or  if  display  1  was  not  requested. 
May  be  saved.) 

Display  3.  Correspondence  among  observers.  3 
Display  4.  Analysis  by  stimuli.  (May  be  saved.)  4 

Display  5.  Distribution  of  ratings.  5 

Display  6.  Stimulus-level  correlation  table.  6 

Display  7.  Condition-level  correlation  table.  7 

Display  8.  Scattergram  of  two  output  measures.  (See  columns  8 
21-25  if  this  display  is  to  be  generated) 

Display  9.  Z-scores  and  associated  statistics.  (May  be  saved.)  9 

Display  10.  Least  squares  ratings  and  associated  statistics.  10 
(May  be  saved.) 

Display  11.  Baseline-adjusted  Z-scores  and  associated  statistics.  11 
(May  be  saved.) 

Display  12.  Baseline-adjusted  least  squares  ratings  and  12 
associated  statistics.  (May  be  saved.) 

Displays  13  -  18.  Principal  component  analysis.  (See  input  line 
15  for  further  requirements  on  this  and  other 
principal  component  analysis  displays.  Note 
that  because  of  the  matrix  manipulations,  the 
elapsed  time  to  perform  the  principal  component 
analysis  increases  geometrically  as  the  number 
of  observers  increases;  i.e.,  on  a  mainframe, 
beware  of  high  costs  with  large  data  sets;  on 
a  PC,  be  patient!) 


Displayed 
Displayed 

Displayed 

Displayed  and  saved 
on  file 

Displayed 

Displayed 

Displayed 

Displayed 

Displayed 
Displayed 

Displayed 

Displayed 


13.  Eigenvalues. 

13 

1 

Displayed 

14.  Component  loadings. 

14 

1 

Displayed 

15.  Component  scores. 

15 

1 

Displayed 

16.  Scattergrams  of  component  loadings  for  component  1 

16 

1 

Displayed 

vs.  loadings  of  other  selected  components. 

17.  Scattergrams  of  component  scores  for  component  1 

17 

1 

Displayed 

vs.  scores  of  other  selected  components. 

18.  Correlations  of  components  with  other  measures. 

18 

1 

Displayed 

Display  19.  Analysis  by  observers. 

19 

1 

Displayed 

Threshold  correlation  coefficient  for  display  8.  Required  only  if 

21-25 

.987 

Plots  statistics 

column  8  is  not  blank.  The  threshold  must  be  greater  than  or 

with  -.987  <  r 

equal  to  0  and  less  than  or  equal  to  1.  A  0  would  mean  no  plots 
would  be  generated;  1  would  mean  all  plots  would  be  generated. 


<.987  in  display  6 


17    "Saved  output"  file  name.  May  NOT  be  the  same  as  the  file  name  on    1-80         SAMPLE. SAV  PC  style  file  name 

line  4.  If  no  such  capturing  is  wanted,  enter  NONE  in  columns  1-4. 
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Output 

RMRATE  output  is  described  in  this  section.  The  out- 
put consists  of  a  listing  of  the  input  parameters,  plus  19 
"displays."  The  displays  consist  of  one  or  more  tables 
or  figures.  Examples  of  an  input  parameter  listing  and 
associated  displays  are  shown  in  appendix  C,  and  an 
example  of  a  more  machine-readable  version  of  display 
4  is  shown  in  appendix  D.  Examples  mentioned  in  this 
section  refer  to  the  appendix  C  and  D  displays. 

Display  1.  Raw  ratings  and  associated  statistics. — 
This  table  presents  three  kinds  of  information:  raw 
ratings,  statistics  for  each  observer  and  each  stimulus, 
and  marks  indicating  which  stimuli  and  observers  are 
to  be  removed. 

The  raw  ratings  are  listed  for  all  observers  and  stimuli 
that  were  included  in  the  input  data  file.  The  ratings  are 
listed  using  the  rating  scale  entered  in  line  7  of  the  input 
controls,  which  may  not  be  the  same  as  the  scale  in 
which  the  ratings  are  recorded  on  the  data  file.  The 
observers  and  stimuli  are  listed  in  the  order  in  which 
they  were  read  in.  The  observers  are  indicated  by  the 
sequence  number  of  the  order  in  which  they  were  read 
in.  The  stimuli  are  indicated  by  sequence  number  un- 
less they  were  given  names  in  line  11  of  the  input,  in 
which  case  the  names  are  shown.  In  addition,  the  stimu- 
lus conditions  are  listed. 

Statistics  are  provided  for  each  observer  and  stimu- 
lus. For  each  observer,  the  following  statistics  are  com- 
puted across  all  stimuli:  (1)  mean  rating,  (2)  median  rat- 
ing, (3)  standard  deviation  of  the  ratings,  (4)  range  of 
the  rating  scale  used  by  the  observer,  (5)  skewness  of  the 
observer's  ratings,  (6)  Pearson  correlation  coefficient  of 
the  observer's  ratings  with  the  mean  ratings  of  all  other 
observers,  and  (7)  significance  of  that  correlation.  For 
each  stimulus,  the  following  statistics  are  computed 
across  all  observers:  (1)  mean  rating,  (2)  median  rating, 
(3)  standard  deviation  of  ratings,  and  (4)  range  of  the 
ratings  assigned  to  that  stimulus  by  all  observers. 

Marks  are  used  to  indicate  which  observers  and 
stimuli  are  to  be  removed  from  the  analyses  to  follow, 
and  why  they  are  to  be  removed.  In  display  1  (appendix 
C),  these  marks  are  found  next  to  observers  3,9,11,  and 
13  and  next  to  stimuli  5,  13,  17,  32,  and  33.  There  are 
four  possible  reasons  for  removal  of  observers  or  stimuli: 
(1)  they  were  excluded  a  priori  by  the  user  (observer  3, 
stimuli  13,  17,  32,  and  33),  (2)  there  were  excessive  blank 
responses  (observer  13,  stimulus  5),  (3)  an  observer's 
ratings  did  not  correlate  well  with  the  mean  ratings  of 
all  other  observers  (observer  9),  and  (4)  an  observer  used 
an  inadequate  range  of  the  rating  scale  (observer  11).  For 
computational  reasons,  an  observer  will  be  removed  by 
default  if  the  range  is  zero,  indicating  that  he/she 
assigned  all  stimuli  the  same  rating. 

Display  2.  Ratings  and  associated  statistics. — This 
table  is  identical  to  the  table  of  display  1  except  that  it 
does  not  include  the  observers  or  stimuli  that  were  re- 
moved. Display  2  thus  displays  only  those  observers  and 
stimuli  that  will  be  included  in  the  analyses  to  follow. 
Display  2  cannot  be  generated  if  no  observers  or  stimuli 
were  removed  and  display  1  was  already  generated  in 


the  same  run.  Notice  in  display  2  that  the  observers  and 
stimuli  retain  their  original  (display  1)  sequence  num- 
bers. Also  note,  by  comparing  displays  1  and  2,  that 
removal  of  observers  and  stimuli  affects  the  statistics  for 
the  remaining  stimuli  and  observers. 

Display  3.  Correspondence  among  observers. — This 
display  is  a  set  of  four  different  kinds  of  tables.  First, 
the  numbers  of  observers  and  stimuli  that  were  removed 
are  listed  according  to  the  reason  for  removal,  and  the 
remaining  totals  are  given.  Second,  the  minimum,  max- 
imum, and  mean  of  the  Pearson  correlation  coefficients 
of  each  observer's  ratings  with  the  mean  ratings  across 
all  other  observers  are  listed,  first  across  all  observers 
and  then  across  all  included  observers.  Comparison  of 
these  two  sets  of  coefficients  indicates  the  effect  of  the 
removals  on  relative  correspondence  among  observers. 
Third,  a  standard  two-way  analysis  of  variance  table 
(stimuli-by-observers)  is  presented  of  the  ratings  of  the 
included  observers.  This  table  facilitates  tests  of  two 
hypotheses:  that  the  observers  did  not  differ  in  their 
ratings,  and  that  the  ratings  of  the  stimuli  do  not  differ. 

Results  of  the  analysis  of  variance  are  used  to  produce 
the  fourth  table,  which  gives  observer-to-observer  and 
group-to-group  reliability  statistics  for  the  ratings.  The 
observer-to-observer  reliability  statistic  gives  the 
expected  correlation  between  any  two  observers  drawn 
from  the  observer  population  at  issue.  The  group-to- 
group  reliability  statistic  indicates  the  expected  corre- 
lation between  group  mean  ratings  for  two  groups  of 
observers  of  the  same  size  as  the  one  at  issue. 

The  analysis  of  variance  and  reliability  tables  are  then 
repeated  for  Z-scores  of  the  ratings. 

Display  4.  Analysis  by  stimuli. — This  table  is  perhaps 
the  focal  point  of  RMRATE  output.  It  presents  the  results 
of  10  different  scalings  of  the  ratings,  plus  5  statistics. 
These  results  are  presented  for  each  stimulus,  each  con- 
dition, all  stimuli  together,  and  all  nonbaseline  stimuli 
together.  The  individual  stimulus  results  are  presented 
in  groups  by  condition.  The  15  scale  values  and  statis- 
tics, listed  in  the  order  they  appear  in  display  4,  are: 

1.  Mean  rating 

2.  Median  rating 

3.  Standard  deviation  of  ratings 

4.  Mean  origin-adjusted  rating  (OAR) 

5.  Mean  Z-score  based  on  ratings  of  all  stimuli 

6.  Mean  least  squares  rating  (LSR)  based  on  ratings 
of  all  stimuli 

7.  Mean  OAR  based  on  ratings  of  baseline  stimuli 

8 .  Mean  Z-score  based  on  ratings  of  baseline  stimuli 

9.  Mean  LSR  based  on  ratings  of  baseline  stimuli 

10.  By-stimulus  SBE 

11.  Standardized  by-stimulus  SBE  (SBE*) 

12.  Skewness  of  ratings 

13.  Skewness  of  Z-scores  based  on  ratings  of  all 
stimuli 

14.  Kurtosis  of  ratings 

15.  Modified  Anderson-Darling  statistic. 

The  10  scaling  procedures  are  described  briefly  here, 
and  in  detail  by  Brown  and  Daniel  (1990).  The  median 
rating  is  the  midpoint  rating  in  the  set  of  ordered  ratings. 
The  mean  rating  is  the  arithmetic  mean,  rounded  to  two 


13 


digits.  An  OAR  is  an  original  rating  minus  the  respec- 
tive observer's  mean  rating.  A  Z-score,  or  standardized 
rating,  is  an  original  rating  minus  the  observer's  mean 
rating,  this  difference  divided  by  the  standard  deviation 
of  the  observer's  ratings.  An  LSR  is  based  on  the  linear 
fit  of  an  observer's  ratings  with  the  mean  ratings 
assigned  by  the  entire  group  of  observers.  Two  sets  of 
OARs,  Z-scores,  and  LSRs  are  provided.  In  the  first  set, 
the  adjustments  are  based  on  ratings  of  all  stimuli  (e.g., 
an  OAR  is  an  original  rating  minus  the  observer's  mean 
rating  across  all  stimuli).  In  the  second  set,  the  adjust- 
ments are  based  on  ratings  of  the  baseline  stimuli  only 
(e.g.,  a  baseline-adjusted  OAR  is  an  original  rating  minus 
the  observer's  mean  rating  across  the  baseline  stimuli). 
The  by-stimulus  SBE  is  the  Scenic  Beauty  Estimate 
introduced  by  Daniel  and  Boster  (1976)  as  the  "by-slide" 
SBE.  Finally,  the  SBE*  is  a  fully  standarized  by-stimulus 
SBE,  where  ratings  of  the  baseline  stimuli  are  used  not 
only  to  determine  the  origin  of  the  SBE  scale  (as  with 
the  original  SBE),  but  also  the  scale's  interval  size. 

The  summary  values,  those  for  each  condition,  all 
stimuli,  and  all  nonbaseline  stimuli,  are  computed  by 
averaging  the  individual  stimulus  values  across  the  rele- 
vant stimuli.  For  example,  the  all-stimuli  SBE  (-8.25,  dis- 
play 4)  is  the  mean  of  the  30  individual  stimulus  SBEs, 
and  the  all-stimuli  standard  deviation  of  ratings  (1.94, 
display  4)  is  the  mean  of  the  30  individual  stimulus 
standard  deviations. 

Display  5.  Distribution  of  ratings. — This  display  con- 
tains two  tables.  The  first  table  lists  the  proportion  of 
included  observers  using  each  possible  range  of  the  rat- 
ing scale.  For  example,  the  ".444"  in  the  first  line  of 
display  5  indicates  that  44%  of  the  observers  recorded 
ratings  that  ranged  8  rating  values  from  the  lowest  to 
highest  rating,  such  as  from  "1"  to  "9",  or  "2"  to  "10". 
This  gives  one  indication  of  the  degree  to  which 
observers  discriminated  among  the  stimuli.  The  second 
table  gives  the  proportion  of  the  ratings  in  each  rating 
category  for  each  stimulus  and  each  condition.  For 
example,  the  ".012"  in  the  upper  left  portion  of  the  table 
indicates  that  1.2%  of  the  respondents  rated  the  aver- 
age baseline  stimulus  a  "1".  Similarly,  22%  of  the 
respondents  rated  stimulus  number  4  a  "2".  These 
proportions  indicate  the  agreement  among  observers 
about  specific  stimuli,  as  well  as  the  distribution  of  rat- 
ings about  the  mean  rating. 

Display  6.  Stimulus-level  correlation  table.  This  table 
lists  Pearson  correlation  coefficients  of  the  scale  values 
of  the  stimuli  for  pairs  of  the  scaling  methods  listed  in 
display  4.  For  example,  the  correlation  of  mean  ratings 
with  mean  Z-scores  across  the  30  included  stimuli  was 
0.988  (display  6),  indicating  that  mean  Z-scores  are 
nearly  a  precise  linear  transformation  of  mean  ratings. 
All  of  the  correlation  coefficients  in  display  6  are  above 
0.985.  Other  data  might  produce  lower  correlation  coeffi- 
cients. Note  that  correlations  with  SBE*s  are  not  listed 
because  they  are  identical  to  those  for  SBEs. 

Display  7.  Condition-level  correlation  table. — This  table 
lists  correlation  coefficients,  similar  to  display  6,  but 
here  conditions  rather  than  stimuli  are  the  cases  of  anal- 
ysis. If  only  one  condition  was  specified  by  the  user  (input 


lines  9  and  10),  this  table  cannot  be  generated.  If  two 
conditions  were  specified,  the  table  can  be  generated, 
but  it  is  meaningless;  with  few  conditions,  the  correla- 
tion coefficients  must  be  quite  high  to  be  significant. 

Display  8.  Scattergram  of  two  output  measures. — This 
display  is  a  set  of  two-dimensional  plots  of  scale  values, 
one  for  each  pair  of  measures  for  which  the  absolute 
value  of  the  correlation  coefficient  in  display  6  is  less 
than  the  threshold  specified  at  the  end  of  input  line  16. 
Normally,  it  is  not  necessary  to  plot  very  high  correla- 
tions, as  values  above  about  0.9  are  only  achieved  when 
there  is  little  or  no  "scatter"  around  a  linear  relation- 
ship. For  the  test  data  set,  however,  a  threshold  level 
of  0.987  was  specified,  so  that  two  plots  (those  compar- 
ing SBE  with  baseline-adjusted  Z-scores  and  least 
squares  ratings)  would  be  printed  for  illustration 
purposes. 

Display  9.  Z-scores  and  associated  statistics. — This 
table  lists  the  Z-scores  for  each  observer-by-stimulus 
combination,  plus  some  statistics  about  the  observers 
and  stimuli.  The  table  is  identical  to  display  2,  except 
that  it  presents  Z-scores  rather  than  ratings.  Displays  10 
through  12  are  identical  in  form  to  display  9,  but  present 
least  squares  ratings,  baseline-adjusted  Z-scores,  and 
baseline-adjusted  least  squares  ratings,  respectively. 
Each  of  these  four  displays  is  perhaps  most  useful  in 
machine-readable  form,  for  use  as  input  to  other  pro- 
grams. That  option  is  requested  on  input  lines  16  and  17. 

Display  13.  Eigenvalues. — This  table  lists  all  eigen- 
values of  the  principal  component  analysis  of  the  ratings 
and  the  percent  of  variance  accounted  for  by  each 
component.  The  percent  of  variance  is  the  eigenvalue 
divided  by  the  number  of  observers.  The  number  of 
eigenvalues  is  equal  to  the  number  of  included 
observers,  in  this  case  9.  If  the  data  set  is  large,  the  num- 
ber of  observers  included  in  the  principal  component 
analysis  may  need  to  be  less  than  the  number  included 
in  the  other  analyses  of  the  run.  If  the  number  of 
observers  must  be  reduced  for  the  principal  component 
analysis,  and  the  user  did  not  select  the  observers  to  be 
included  on  input  control  line  15,  the  maximum  possi- 
ble number  of  observers  is  randomly  selected  by 
RMRATE. 

No  reduction  in  number  of  observers  was  necessary 
for  the  test  data  set.  The  eigenvalues  for  the  test  data  set 
vary  from  3.634  to  0.2926  (display  13).  The  first  prin- 
cipal component  accounts  for  40%  of  the  variance  in  the 
ratings  of  the  9  included  observers.  The  second  and  third 
components  account  for  an  additional  14%  and  12%, 
respectively,  suggesting  that  there  may  be  some  consis- 
tent variations  in  preference  among  subsets  of  the 
observers. 

Display  14.  Principal  Component  loadings. — This 
table  lists  the  component  loadings  of  the  selected  com- 
ponents for  each  included  observer.  Unless  the  desired 
components  were  specified  by  the  user  on  input  control 
line  15,  RMRATE  prints  loadings  for  all  components 
with  an  eigenvalue  of  1.0  or  more.  This  default  was 
chosen  for  the  test  data  set  (notice  the  blank  line  15  of 
the  input  controls,  appendix  B),  resulting  in  the  listing 
of  loadings  for  three  principal  components  (display  14). 


14 


These  loadings  suggest  that  several  observers  have  per- 
ceptions that  diverge  somewhat  from  the  aggregate  or 
consensus  preference.  Specifically,  observers  7  and  8 
have  their  strongest  loading  on  component  2  rather  than 
component  1,  and  observer  5's  strongest  loading  is  on 
component  3.  Observers  2  and  10  also  have  moderately 
strong  loadings  on  components  2  and  3,  respectively. 
This  suggests  that  components  2  and  3  may  reflect  per- 
ceptual dimensions  of  the  stimuli  to  which  these 
observers  have  divergent  responses. 

Display  15.  Principal  Component  scores. — This  table 
lists  component  scores  of  the  selected  components  for 
each  stimulus.  The  scores  for  the  three  components 
selected  for  the  test  data  set  are  shown  in  display  15. 
The  scores  of  stimuli  on  the  first  principal  component 
represent  the  majority  or  aggregate  consensus  of  the 
observers.  They  will  almost  always  correlate  very  closely 
with  the  scale  values  obtained  from  the  other  analyses 
in  RMRATE.  In  fact,  the  scores  for  the  first  principal 
component  may  be  viewed  as  an  additional  option  for 
scaling  the  stimuli.  Note,  however,  that  the  direction  of 
the  scale  for  the  component  scores  may  occasionally  be 
opposite  to  that  of  the  other  scales.  That  is,  high  com- 
ponent scores  may  correspond  to  low  ratings  and  low 
component  scores  may  correspond  to  high  ratings.  When 
this  occurs,  the  majority  of  observers  will  have  negative 
loadings  on  component  1  in  display  14. 

The  scores  on  the  other  components  may  be  helpful 
for  interpreting  variations  in  perceptions  within  the 
group.  This  is  done  by  examining  the  stimuli  that  have 
the  highest  and  lowest  scores  on  each  component.  For 
example,  on  component  2  the  highest  scores  are  for 
stimuli  4  and  11.  Observers  with  strong  positive  load- 
ings on  component  2  (i.e.,  observers  2  and  8)  gave  higher 
than  average  ratings  to  these  stimuli,  and  lower  than 
average  ratings  to  stimuli  with  low  scores  on  component 
2  (e.g.,  stimuli  16  and  20).  On  the  other  hand,  observer 
7's  strong  negative  loading  on  component  2  means  that 
he  or  she  gave  higher  than  average  ratings  to  stimuli 
16  and  20,  and  lower  than  average  ratings  to  stimuli  4 
and  11. 

To  interpret  variations  in  preference,  the  investigator 
must  be  able  to  discern  features  that  differentiate  stimuli 
with  high  loadings  on  a  component  from  stimuli  with 
low  loadings  on  the  same  component.  Frequently  this 
can  be  done  with  an  "eyeball"  examination  of  the 
stimuli.  If  measures  of  physical  features  of  the  stimuli 
are  available,  these  can  be  correlated  with  or  regressed 
on  the  component  scores  to  obtain  a  more  rigorous 
interpretation  of  each  component.  Schroeder  (1987)  pro- 
vides a  detailed  example  of  this  procedure. 

Display  16.  Scattergram  of  component  loadings. — 
This  display  consists  of  a  set  of  two-dimensional  plots, 
each  showing  the  loadings  of  component  1  plotted 
against  the  loadings  of  one  of  the  other  selected  compo- 
nents. For  the  test  data  set,  where  three  components  were 
selected,  this  resulted  in  two  plots.  The  plots  of  compo- 
nent loadings  are  provided  as  an  aid  to  identifying 
observers  that  have  high  and  low  loadings  on  each 


principal  component,  and  for  obtaining  an  overall  visual 
display  of  how  observers  are  distributed  over  the  dimen- 
sions of  the  principal  component  solution. 

Each  observer  is  indicated  on  the  plots  by  a  letter. 
RMRATE  assigns  letters  sequentially  to  observers,  in  the 
order  in  which  observers  were  entered  in  the  data  file. 
First,  the  set  of  upper-case  letters  is  used,  followed  by 
the  set  of  lower-case  letters.  If  additional  symbols  are 
needed,  the  program  repeats  this  procedure.  If  more  than 
one  observer  is  plotted  in  the  same  cell,  each  is  assigned 
the  same  letter.  Each  plot  is  preceded  by  a  list  of  the 
plotted  observers  and  their  respective  symbols  and  load- 
ings for  the  two  plotted  components. 

Display  17.  Scattergram  of  component  scores. — This 
display  consists  of  a  set  of  two-dimensional  plots,  each 
showing  the  scores  of  component  1  plotted  against  those 
of  one  of  the  other  selected  components.  Again,  for  the 
test  data  set,  this  resulted  in  the  printing  of  2  plots.  The 
plots  of  component  scores  are  provided  as  an  aid  to  iden- 
tifying stimuli  that  have  high  and  low  scores  on  each 
component,  and  for  obtaining  an  overall  visual  display 
of  how  stimuli  are  distributed  over  the  dimensions  of 
the  principal  component  solution. 

As  with  the  plots  of  component  loadings,  the  com- 
ponent score  plots  indicate  the  stimuli  by  upper-  and 
lower-case  letters.  Each  plot  is  preceded  by  a  list  of  the 
stimuli  and  their  respective  symbols  and  scores  for  the 
plotted  components. 

Display  18.  Correlation  of  component  scores  with 
other  measures  of  interest.— This  table  gives  Pearson 
correlation  coefficients  of  the  scores  for  the  selected  com- 
ponents with  the  scale  values  for  the  scaling  methods 
included  in  display  4.  For  example,  the  correlation 
coefficient  of  the  scores  for  component  1  with  the  mean 
ratings  for  the  stimuli  is  0.981  (display  18).  The  corre- 
lation coefficients  in  display  18  indicate  that,  as  noted 
above,  the  scores  on  component  1  correspond  closely  to 
the  other  aggregate  scale  values  calculated  by  RMRATE. 
The  remaining  components  have  negligible  correlations 
with  the  other  scale  values.  This  is  to  be  expected,  since 
these  components  are  constrained  to  be  statistically 
independent  of  component  1. 

Display  19.  Analysis  by  observers. — This  table 
presents  by-observer  SBEs,  plus  mean  and  median  rat- 
ings and  additional  statistics,  for  each  individual 
observer.  This  is  done  for  all  stimuli,  all  nonbaseline 
stimuli,  and  for  each  condition.  Each  observer's  value 
represents  his  or  her  responses  to  the  stimuli  in  the 
respective  set  of  stimuli.  For  example,  observer  l's  mean 
rating  across  all  stimuli  was  7.30,  and  that  observer's 
mean  rating  across  the  baseline  stimuli  was  7.11  (dis- 
play 19).  Similarly,  the  standard  deviations  of  observer 
l's  ratings  of  all  stimuli  and  the  baseline  stimuli  were 
0.88  and  0.93,  respectively  (display  19).  Note  that  in  dis- 
play 19  the  values  for  all  stimuli,  all  nonbaseline  stimuli, 
and  the  conditions  (MIXPRE  and  MIXPOST)  are  the  aver- 
ages of  the  individual  observer  values.  Thus,  the  all- 
stimuli  standard  deviation  (1.74)  is  the  mean  of  the  9 
observers'  standard  deviations. 
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APPENDIX  A 


SAMPLE  DATA  FILE:  SAMPLE.DAT 
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APPENDIX  B 
SAMPLE  INPUT  PARAMETERS:  SAMPLE. RUN 


SAMPLE  RUNSTREAM 
13  35 
1 

SAMPLE.DAT 
(T22,  1312) 
10 
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APPENDIX  C 
SAMPLE  OUTPUT  (SAMPLE. OUT) 

RMRATE  06-07-90  13:49:47 

RMRATE  is  described  in  GTR  RM  195.  "Analysis  of  Ratings:  A  Guide  to  RMRATE,"  by  Thomas  C.  Brown,  Terry  C.  Daniel,  Herbert  W.  Schroeder 
and  Glen  E.  Brink.  More  detail  on  the  scaling  procedures  and  statistics  of  RMRATE  is  found  in  Research  Paper  RM-293,  "Analysis  of  Ratings: 
Concepts  and  Methods,"  by  Brown  and  Daniel.  Both  reports  are  available  from  Rocky  Mountain  Forest  and  Range  Experiment  Station  Publica- 
tions, 240  W.  Prospect  St.,  Fort  Collins,  CO  80526. 

This  version  of  RMRATE  requires  560K  available  bytes  of  RAM  (memory),  plus  a  math  coprocessor. 

NUMBER  OF  OBSERVERS:    13,  NUMBER  OF  STIMULI:  35 
(RESULTING  MEMORY  ENVIRONMENT:  MAXOBS  =    14,  MAXFAC  =  14) 

INPUT  IS  BY  STIMULUS 

RATINGS  WILL  BE  READ  FROM  SAMPLE.DAT 
INPUT  FORMAT:  (T22,  1312) 

NUMBER  OF  RATING  VALUES  IN  THE  RATING  SCALE:  10 

RATING  VALUES  (NOT  RATED,  LEAST  PREFERRED,  .  .  .  ,  MOST  PREFERRED):     01      23456789  10 
CORRESPONDING  INPUT  VALUES  -1  0123456789 

NUMBER  OF  CONDITIONS  (INCLUDING  BASELINE):  3 

CONDITION    0:    BASELINE      1      23456789  10 

CONDITION    1:    MIXPRE       11    12    13    14    15    16    17    18    19    20    21    22    23  24 

CONDITION    2:    MIXPOST     (ALL  OTHER  STIMULI) 

STIMULI  NAMES:    NO  NAMES 

NUMBER  OF  BLANKS  TO  CAUSE  REMOVAL  OF  AN  OBSERVER:  1 

NUMBER  OF  BLANKS  TO  CAUSE  REMOVAL  OF  A  STIMULUS:  1 

CORRELATION  COEFFICIENT  CUTOFF  FOR  REMOVAL  OF  AN  OBSERVER:  -.700 

MINIMUM  RANGE  IN  RATINGS  FOR  INCLUSION  OF  AN  OBSERVER:  1 

EXCLUDED  OBSERVERS: 
3  -  3 

EXCLUDED  STIMULI: 
13  -  13 
17  -  17 
32  -  33 

PRINCIPAL  COMPONENT  ANALYSIS:    NO  SPECIFIED  NUMBER  OF  COMPONENTS  TO  PRINT 

PRINCIPAL  COMPONENT  ANALYSIS:    DEFAULT  MINIMUM  EIGENVALUE  FOR  PRINTING  OF  COMPONENTS  (1.0) 

PRINCIPAL  COMPONENT  ANALYSIS:    ALL  OBSERVERS  WILL  BE  INCLUDED 

OUTPUT  DISPLAYS:      1      2     3     4     5     6     7     8     9    10    1 1    12    13    14    15    16    17    18  19 
OPTIONS:      1112  111111111111111 

PLOT  OUTPUT  MEASURES  WITH  CORR.  LESS  THAN  .987 

REQUESTED  OUTPUT  WILL  BE  SAVED  ON  SAMPLE. SAV 
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SAMPLE  RUNSTREAM 

DISPLAY  1.  RAW  RATINGS  AND  ASSOCIATED  STATISTICS 
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1 

9 

wax* 

6 

4 

20 

MIXPRE 

4.77 

4.0 

2.01 

6.0 

7 

3 

5 

4 

7 

3 

7 

2 

8 

3 

6 

3 

21 

MIXPRE 

4.62 

4.0 

2.33 

7.0 

7 

1 

6 

5 

4 

1 

8 

3 

8 

3 

6 

4 

22 

MIXPRE 

4.69 

5.0 

1.89 

6.0 

8 

2 

5 

5 

4 

2 

4 

4 

8 

3 

6 

5 

23 

MIXPRE 

5.77 

6.0 

2.39 

7.0 

9 

6 

4 

8 

4 

4 

5 

9 

9 

2 

6 

3 

24 

MIXPRE 

7.15 

8.0 

1.41 

4.0 

8 

9 

7 

9 

8 

5 

8 

8 

5  T 

6 

6 

8 

25 

MIXPOST 

4.92 

6.0 

2.10 

7.0 

7 

6 

4 

4 

6 

4 

1 

7 

8 

3 

6 

2 

26 

MIXPOST 

5.31 

5.0 

1.75 

5.0 

7 

4 

7 

8 

5 

3 

4 

3 

7 

4 

6 

7 

27 

MIXPOST 

4.31 

5.0 

2.06 

6.0 

7 

5 

4 

7 

5 

2 

1 

1 

6 

5 

6 

3 

28 

MIXPOST 

5.85 

5.0 

1.82 

5.0 

8 

5 

5 

7 

5 

3 

8 

8 

8 

3 

6 

5 

29 

MIXPOST 

5.15 

5.0 

2.15 

7.0 

8 

5 

7 

5 

7 

2 

1 

7 

7 

4 

6 

3 

30 

MIXPOST 

5.54 

6.0 

1.13 

4.0 

6 

6 

6 

7 

5 

3 

7 

6 

6 

5 

6 

4 

31 

MIXPOST 

5.31 

6.0 

1.93 

7.0 

7 

5 

8 

6 

5 

2 

1 

7 

6 

5 

6 

6 

32 

!  MIXPOST 

4.69 

4.0 

2.02 

6.0 

8 

5 

8 

4 

4 

2 

4 

2 

7 

4 

6 

4 

33 

!  MIXPOST 

4.62 

4.0 

2.53 

7.0 

8 

6 

4 

6 

2 

2 

7 

2 

9 

2 

6 

4 

34 

MIXPOST 

5.38 

4.0 

1.85 

5.0 

8 

4 

8 

4 

4 

3 

4 

8 

7 

4 

6 

6 

35 

MIXPOST 

4.69 

4.0 

2.29 

7.0 

9 

6 

4 

4 

4 

3 

4 

2 

9 

2 

6 

5 

# 

4.68 
5.00 
1.27 
5.0 
-.515 
.402 
.009 

4 
3 
5 
5 
0 

5 
4 
6 
3 
5 

7 
5 
7 
4 
5 

6 
6 
6 
2 
4 

4 
5 
6 
6 
6 

4 
4 
5 
5 
5 

5 
3 
2 
4 
3 


!    =  EXCLUDED  BY  USER 

&  =  REMOVED  BECAUSE  OF  POOR  CORRELATION  WITH  GROUP 


=  REMOVED  BECAUSE  OF  EXCESSIVE  BLANK  RATINGS 
=  REMOVED  BECAUSE  OF  POOR  RANGE  OF  RATINGS 
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SAMPLE  RUNSTREAM 

DISPLAY  2.  RATINGS  AND  ASSOCIATED  STATISTICS 


06-07-90  13:49:47 
PAGE  1 


STD.  OBSERVERS: 


STIMULUS 

CONDITION 

MEAN 

MED. 

DEV. 

RANGE 

1 

2 

4 

5 

6 

7 

8 

10 

12 

MEAN 

7.30 

4.93 

5.80 

4.60 

2.93 

4.90 

5.40 

3.90 

4.07 

MEDIAN 

7  nn 
/  .uu 

4  Rn 

H.OU 

r  Rn 

O.OU 

4  nn 
^.uu 

q  nn 

O.UU 

4  Rn 
H  .ou 

r  nn 

O.UU 

a  nn 

a  nn 
4.UU 

STANDARD  DEVIATION 

ftft 

1  HQ 

i  .oy 

1  R1 
I  .O  I 

1  9R 
I  .^o 

1  Rn 

l  .OU 

9  dq 

<l  .  *fO 

9  7n 

c..  1  u 

1  q9 

I  .o£ 

9  nq 
^.uo 

RANGE 

q  n 
o.u 

fl  n 

o.u 

r  n 

o.u 

r  n 
o.u 

r  n 
o.u 

r  n 

O.U 

q  n 

r  n 
o.u 

r  n 
o.u 

SKEWNESS 

mo 
.oUo 

qft7 

qRq 
.oOo 

.OOC7 

1  Q9R 

nfio, 

—  .UOc? 

1  9fl 

qRn 

.oOU 

R7Q 

.O  /  3 

CORRELATION  W/  GROUP 

.479 

.573 

.641 

.361 

.647 

.299 

.348 

.518 

.544 

SIGNIFICANCE  OF  R 

.004 

.000 

.000 

.025 

.000 

.054 

.030 

.002 

.001 

1 

BASELINE 

R  R7 
O.O  f 

R  fl 
D.U 

1  rr 

r  n 
o.u 

7 

7 

c 
o 

A 

q 
o 

D 

□ 

A 
H- 

C 
D 

2 

BASELINE 

a  n 
h  .u 

r  n 
o.u 

7 
i 

*+ 

c 
o 

R 
O 

o 
c. 

A 
H 

q 
O 

C 

O 

1 
I 

3 

BASELINE 

T  7ft 
o.  /  O 

q  n 

O.U 

1  flQ 

t  n 
o.u 

ft 
o 

*t 

q 
o 

q 
o 

q 
o 

q 
o 

c 
o 

q 
o 

H 

4 

BASELINE 

4.67 

5.0 

2.29 

6.0 

7 

7 

5 

3 

2 

2 

8 

5 

3 

6 

BASELINE 

4.78 

4.0 

1.79 

4.0 

7 

7 

5 

4 

3 

3 

7 

3 

4 

7 

BASELINE 

4  qq 

4  n 

A  O 

7 
1 

q 
o 

c 
o 

A 

q 

O 

A 

*t 

q 
o 

R 

o 

8 

BASELINE 

fi  99 

7  n 

7  n 
/  .u 

o 
o 

g 

7 

4 

2 

a 
o 

c 
u 

R 
o 

7 

9 

BASELINE 

a  nn 

a  n 

H.U 

1  Rft 

*r -U 

c 
o 

/i 

R 

o 

q 

O 

o 

r 

o 

c 
o 

O 
c. 

q 

10 

BASELINE 

8.44 

9.0 

1.24 

4.0 

9 

8 

9 

6 

9 

9 

10 

7 

9 

11 

MIXPRE 

4.22 

4.0 

2.39 

8.0 

6 

5 

4 

3 

1 

3 

9 

5 

2 

12 

MIXPRE 

q  f\7 
O.O  / 

a  n 
f  .u 

1  sn 

l  .OU 

o.u 

c 
o 

*+ 

c 
o 

r 

o 

•) 

4 

p 

q 
o 

o 

14 

MIXPRE 

A  OO 

1  -U 

1  dft 

I  .HO 

o.u 

7 

4 

r 

o 

4 

q 
o 

5 

R 
o 

q 
o 

2 

15 

MIXPRE 

o.uu 

o.u 

o  Rn 

7  fi 
/  .u 

7 

q 
o 

7 

r 
o 

0 

7 

fl 
o 

R 
O 

•| 

16 

MIXPRE 

5.33 

6.0 

2.29 

6.0 

8 

4 

8 

5 

6 

7 

2 

6 

2 

18 

MIXPRE 

4.56 

4.0 

1.59 

5.0 

8 

4 

5 

4 

4 

6 

3 

4 

3 

19 

MIXPRE 

q  RQ 

a  n 

O  10 

7  n 
/  .u 

7 

A 

H 

4 

H 

q 
o 

2 

s 

-| 

2 

4 

20 

MIXPRE 

A  *M 

q  n 

0  Ofi 

r  n 
o.u 

7 

•5 
O 

4 

7 

3 

7 

2 

3 

3 

21 

MIXPRE 

a  nn 

*t  .u 

o  An 

^.*tU 

7  n 
/  .u 

7 

■1 
1 

R 

o 

t 

1 

3 

q 
o 

q 

O 

4 

22 

MIXPRE 

4  11 

*T.  U 

1  fti 

6.0 

s 

2 

5 

4 

2 

4 

4 

3 

5 

23 

MIXPRE 

5.56 

5.0 

2.60 

7.0 

9 

6 

8 

4 

4 

5 

9 

2 

3 

24 

MIXPRE 

7.67 

8.0 

1.32 

4.0 

8 

9 

9 

8 

5 

8 

8 

6 

8 

25 

MIXPOST 

4.44 

4.0 

2.19 

6.0 

7 

6 

4 

6 

4 

1 

7 

3 

2 

26 

MIXPOST 

5.00 

4.0 

1.87 

5.0 

7 

4 

8 

5 

3 

4 

3 

4 

7 

27 

MIXPOST 

4.00 

5.0 

2.35 

6.0 

7 

5 

7 

5 

2 

1 

1 

5 

3 

28 

MIXPOST 

5.78 

5.0 

2.05 

5.0 

8 

5 

7 

5 

3 

8 

8 

3 

5 

29 

MIXPOST 

4.67 

5.0 

2.40 

7.0 

8 

5 

5 

7 

2 

1 

7 

4 

3 

30 

MIXPOST 

5.44 

6.0 

1.33 

4.0 

6 

6 

7 

5 

3 

7 

6 

5 

4 

31 

MIXPOST 

4.89 

5.0 

2.09 

6.0 

7 

5 

6 

5 

2 

1 

7 

5 

6 

34 

MIXPOST 

5.00 

4.0 

1.87 

5.0 

8 

4 

4 

4 

3 

4 

8 

4 

6 

35 

MIXPOST 

4.33 

4.0 

2.18 

7.0 

9 

6 

4 

4 

3 

4 

2 

2 

5 

20 
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DISPLAY  3.  CORRESPONDENCE  AMONG  OBSERVERS 
REMOVALS 


PAGE  1 


NUMBER 
IN 

RAW  DATA 


EXCLUDED 
BY 
USER 


REMOVED, 
TOO  MANY 
BLANKS 


REMOVED, 
POOR  CORR. 
WITH  GROUP 


REMOVED, 
POOR  RANGE 
OF  RATINGS 


OBSERVERS 
STIMULI 


13 
35 


1 


1 


CORRELATION  OF  INDIVIDUAL  OBSERVERS  WITH  GROUP  (RATINGS) 


ALL  OBSERVERS  AND  STIMULI: 

ONLY  INCLUDED  OBSERVERS  AND  STIMULI: 

3a.    ANALYSIS  OF  VARIANCE  (RATINGS) 

GRAND  MEAN:  4.9 
SOURCE  OF  VARIATION  DF 


MIN 

-.738 
.299 


MEAN 

.324 
.490 


MAX 

.660 
.647 


SS 


MS 


REMAINDER 
INCLUDED 


9 
30 


SIGNIFICANCE  LEVEL 


MEAN 

1 

6404.54 

OBSERVERS 

8 

373.963 

46.7454 

19.1594 

501996E-21 

STIMULI 

29 

306.463 

10.5677 

4.33135 

.106622E-09 

OBS.  X  STIM. 

232 

566.037 

2.43981 

TOTAL 

270 

7651.00 

4a.  RELIABILITY 

OBSERVER  TO  OBSERVER 
GROUP  TO  GROUP 

3b.    ANALYSIS  OF  VARIANCE  (Z-SCORES) 
GRAND  MEAN: 

SOURCE  OF  VARIATION  DF 


.270 
.769 


SS 


MS 


SIGNIFICANCE  LEVEL 


MEAN 

1 

.104825E- 

11 

OBSERVERS 

8 

.117369E- 

11 

.146711E-12 

.214745E-12 

1 .00000 

STIMULI 

29 

102.501 

3.53451 

5.17357 

.216771 E 

OBS.  X  STIM. 

232 

158.499 

.683186 

TOTAL 

270 

261 .000 

4b.  RELIABILITY 


OBSERVER  TO  OBSERVER 
GROUP  TO  GROUP 


.317 
.807 


21 
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DISPLAY  4.  ANALYSIS  BY  STIMULI 
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TRANSFORMS  BASED  ON  OBSERVERS  RESPONSES  TO 


INDICATION     OF  NORMALITY 


RATING 


ALL  INCLUDED  STIMULI 


BASELINE  STIMULI  - 


SKEWNESS 


KURTOSIS  MOD.  A-D 


STIMULUS 

MEAN 

MEDIAN 

STD.  DEV. 

OAR 

Z 

LSR 

OAR 

Z 

LSR 

SBE 

SBE* 

RATING 

Z 

RATING 

RATING 

ALL  STIM. 

4.87 

4.8 

1 .94 

.00 

.00 

4.87 

O/l 

—  ,<i4 

-.06 

5.04 

-8.25 

-16.12 

.  1 00 

.116 

-1 .265 

.561 

BASELINE 

5.11 

5.1 

1 .69 

.*:4 

no 

.uy 

A  QC 

nn 
-UU 

.UU 

5.1 1 

.UU 

.UU 

.029 

.220 

-1 .335 

.582 

1 

5.67 

6.0 

1.66 

.80 

.32 

5.11 

.56 

.24 

5.39 

18.62 

36.37 

-.260 

.077 

-1.501 

.454 

2 

4.11 

4.0 

1.90 

-.76 

-.32 

4.64 

-1.00 

-.36 

4.67 

-36.00 

-70.31 

-.140 

.011 

-1.307 

.179 

3 

3.78 

3.0 

1.09 

-1.09 

-.73 

4.40 

-1.33 

-.81 

4.13 

-45.08 

-88.03 

.891 

-.238 

-.784 

1.066 

4 

4.67 

5.0 

2.29 

-.20 

-.17 

4.78 

-.44 

-.24 

4.79 

-15.35 

-29.97 

.142 

.279 

-1.793 

.466 

6 

4.78 

4.0 

1.79 

-.09 

-.12 

4.81 

-.33 

-.19 

4.87 

-10.12 

-19.77 

.295 

.732 

-1.869 

.813 

7 

4.33 

4.0 

1.41 

-.54 

-.31 

4.68 

-.78 

-.38 

4.68 

-25.84 

-50.47 

.655 

.188 

-1.111 

.598 

8 

6.22 

7.0 

2.22 

1.35 

.71 

5.36 

1.11 

.59 

5.85 

37.42 

73.07 

-.550 

-.071 

-1.070 

.312 

9 

4.00 

4.0 

1.58 

-.87 

-.67 

4.43 

-1.11 

-.70 

4.27 

-39.22 

-76.60 

.000 

-.059 

-1.791 

.418 

10 

8.44 

9.0 

1.24 

3.57 

2.07 

6.31 

3.33 

1.85 

7.36 

115.57 

225.70 

-.770 

1.066 

-.790 

.930 

ALL  NON-BASELINE 

STIMULI 

4.77 

4.6 

2.04 

—.10 

— .U4 

A  QA 

—  .OH 

no 

D.U1 

—  11 .79 

oo  no 

.  1  O  I 

.U  Id 

— 1  .doo 

ceo 

MIXPRE 

4.71 

4.6 

2.05 

1  c 
—.ID 

no 

a  on 

—  .4U 

A  OC 

4.yo 

—  I  O.CH 

OC  CO 

■  £LO\> 

1    1  A  Q 
—  1  . 1  40 

C07 
OO  / 

11 

4.22 

4.0 

2.39 

-.65 

-.52 

4.52 

-.89 

-.53 

4.42 

-30.92 

-60.38 

.552 

.748 

-.710 

.272 

12 

3.67 

4.0 

1.80 

-1.20 

-.67 

4.41 

-1.44 

-.68 

4.28 

-50.30 

-98.24 

-.013 

.272 

-1.647 

.297 

14 

4.22 

4.0 

1.48 

-.65 

-.40 

4.60 

-.89 

-.44 

4.58 

-31.85 

-62.20 

.280 

-.215 

-.935 

.341 

15 

5.00 

5.0 

2.50 

.13 

.03 

4.85 

-.11 

.05 

5.15 

-6.62 

-12.93 

-.341 

-.395 

-1.629 

.478 

16 

5.33 

6.0 

2.29 

.46 

.45 

5.19 

.22 

.30 

5.51 

6.17 

12.06 

-.308 

-.263 

-1.551 

.375 

18 

4.56 

4.0 

1.59 

-.31 

-.10 

4.80 

-.56 

-.21 

4.90 

-17.98 

-35.11 

.988 

.289 

-.266 

.774 

19 

3.89 

4.0 

2.32 

-.98 

-.63 

4.44 

-1.22 

-.71 

4.30 

-40.63 

-79.35 

.549 

.805 

-1.163 

.495 

20 

4.33 

3.0 

2.06 

-.54 

-.24 

4.65 

-.78 

-.24 

4.80 

-23.79 

-46.46 

.440 

.887 

-1.807 

1.124 

21 

4.00 

4.0 

2.40 

-.87 

-.55 

4.46 

-1.11 

-.57 

4.47 

-38.29 

-74.78 

.290 

.347 

-1.313 

.318 

22 

4.11 

4.0 

1.83 

-.76 

-.38 

4.60 

-1.00 

-.43 

4.64 

-36.00 

-70.30 

.722 

.215 

-.332 

.492 

23 

5.56 

5.0 

2.60 

.69 

.39 

5.14 

.44 

.33 

5.55 

17.90 

34.95 

.171 

-.168 

-1.714 

.378 

24 

7.67 

8  0 

1.32 

2.80 

1.64 

5.97 

2.56 

1.54 

6.94 

88.97 

173.75 

-.896 

.242 

-.654 

1.099 

MIXPOST 

4.84 

4.7 

2.04 

-.03 

.02 

4.89 

-.27 

-.03 

5.07 

-9.36 

-18.28 

.034 

-.140 

-1.357 

.571 

25 

4.44 

4.0 

2.19 

-.43 

-.20 

4.73 

-67 

-.22 

4.78 

-24.00 

-46.86 

-.189 

-.033 

-1.647 

.356 

26 

5.00 

4.0 

1.87 

.13 

.13 

4.99 

-.11 

.03 

5.18 

-2.26 

-4.42 

.407 

.564 

-1.658 

.642 

27 

4.00 

5.0 

2.35 

-.87 

-.31 

4.69 

-1.11 

-.39 

4.64 

-39.86 

-77.84 

-.052 

-.248 

-1.736 

.469 

28 

5.78 

5.0 

2.05 

.91 

.44 

5.15 

.67 

.41 

5.62 

22.97 

44.86 

-.123 

-.396 

-1.765 

.629 

29 

4.67 

5.0 

2.40 

-.20 

02 

4.84 

-.44 

.05 

5.12 

-16.28 

-31.79 

-.086 

.285 

-1.582 

.243 

30 

5.44 

6.0 

1.33 

.57 

.23 

5.04 

.33 

.17 

5.29 

11.25 

21.97 

-.457 

-1.29 

1 

-1.167 

.394 

31 

4.89 

5.0 

2.09 

02 

.04 

4.91 

-.22 

-.02 

5.07 

-10.28 

-20.08 

-.749 

-.712 

-1.007 

.711 

34 

5.00 

4.0 

1.87 

.13 

.04 

4.89 

-.11 

-.03 

5.09 

.96 

1.88 

.713 

.013 

-1.331 

1.205 

35 

4.33 

4.0 

2.18 

-.54 

-.19 

4.74 

-.78 

-.30 

4.81 

-26.77 

-52.27 

.844 

.556 

-.318 

.494 

22 


SAMPLE  RUNSTREAM 

DISPLAY  5.  DISTRIBUTION  OF  RATINGS 
1.    PROPORTIONS  OF  OBSERVERS  USING  RANGE  OF  RATINGS 
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RANGE: 
PROPORTION: 


9 
.11 1 


8 
.444 


7 
.000 


6 
.111 


5 
.222 


4 
.000 


3 
.111 


2 
.000 


1 

.000 


0 
.000 


2.    PfiOPOjiILQNS_OF_RESPONSES  IN  EACH  OF  THE  FOLLOWING  RATING  CATEGORIES: 


1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

.  STIM. 

.044 

.093 

.163 

.181 

.148 

.100 

.130 

.093 

.044 

.004 

3ELINE 

.012 

.074 

.210  . 

.160 

.123 

.123 

.148 

.062 

.074 

.012 

1 

.000 

.000 

.111 

.222 

.000 

.333 

.222 

.111 

.000 

.000 

2 

.111 

.111 

.111 

.222 

.222 

.111 

.111 

.000 

.000 

.000 

3 

.000 

.000 

.556 

.222 

.111 

.111 

.000 

.000 

.000 

.000 

4 

.000 

.222 

.222 

.000 

.222 

.000 

.222 

.111 

.000 

.000 

6 

.000 

.000 

.333 

.222 

.111 

.000 

.333 

.000 

.000 

.000 

7 

.000 

.000 

.333 

.333 

.111 

.111 

.111 

.000 

.000 

.000 

8 

.000 

.111 

.000 

.111 

.111 

.111 

.222 

.222 

.111 

.000 

9 

.000 

.222 

.222 

.111 

.222 

.222 

.000 

.000 

.000 

.000 

10 

.000 

.000 

.000 

.000 

.000 

.111 

.111 

.111 

.556 

.111 

PRE 

.056 

.120 

.157 

194 

.139 

.074 

.093 

.1 20 

.046 

.000 

11 

.111 

.111 

.222 

.111 

.222 

.111 

.000 

.000 

.111 

.000 

12 

.111 

.222 

.111 

.222 

.111 

.222 

.000 

.000 

.000 

.000 

14 

.000 

.111 

.222 

.222 

.333 

.000 

.111 

.000 

.000 

.000 

15 

.111 

.111 

.111 

.000 

.222 

.000 

.333 

.111 

.000 

.000 

16 

.000 

.222 

.000 

.111 

.111 

.222 

.111 

.222 

.000 

.000 

18 

.000 

.000 

.222 

.444 

.111 

.111 

.000 

.111 

.000 

.000 

19 

.111 

.222 

.111 

.333 

.000 

.000 

.111 

.111 

.000 

.000 

20 

.000 

.111 

.444 

.111 

.000 

.000 

.333 

.000 

.000 

.000 

21 

.222 

.000 

.222 

.222 

.111 

.000 

.111 

.111 

.000 

.000 

22 

.000 

.222 

.111 

.333 

.222 

.000 

.000 

.111 

.000 

.000 

23 

.000 

.111 

.111 

.222 

.111 

.111 

.000 

.111 

.222 

.000 

24 

.000 

.000 

.000 

.000 

.111 

.111 

.000 

.556 

.222 

.000 

POST 

.062 

.074 

.123 

.185 

.185 

.111 

.160 

.086 

.012 

.000 

25 

.111 

.111 

.111 

.222 

.000 

.222 

.222 

.000 

.000 

.000 

26 

.000 

.000 

.222 

.333 

.111 

.000 

.222 

.111 

.000 

.000 

27 

.222 

.111 

.111 

.000 

.333 

.000 

.222 

.000 

.000 

.000 

28 

.000 

.000 

.222 

.000 

.333 

.000 

.111 

.333 

.000 

.000 

29 

.111 

.111 

.111 

.111 

.222 

.000 

.222 

.111 

.000 

.000 

30 

.000 

.000 

.111 

.111 

.222 

.333 

.222 

.000 

.000 

.000 

31 

.111 

.111 

.000 

.000 

.333 

.222 

.222 

.000 

.000 

.000 

34 

.000 

.000 

.111 

.556 

.000 

.111 

.000 

.222 

.000 

.000 

35 

.000 

.222 

.111 

.333 

.111 

.111 

.000 

.000 

.111 

.000 

23 


SAMPLE  RUNSTREAM 

DISPLAY  6.  STIMULUS-LEVEL  CORRELATION  DISPLAY 


06-07-90  13:49:47 
PAGE  1 


STIMULUS 


RATING 


TRANSFORMS  BASED  ON  OBSERVERS'  RESPONSES  TO 
1)  ALL  INCLUDED  STIMULI    2)  BASELINE  STIMULI 

Z  LSR  Z  LSR  SBE 


ALL  STIM. 
Z-SCORE 

ALL  STIM. 
LSR 

BL  STIMULI 

Z-SCORE 

BL  STIMULI 
LSR 


.988 


.989 


.988 


.987 


.999 


.998 


.998 


.994 
.995 


.999 


SBE 


.999 


.987 


.987 


.986 


.986 


SAMPLE  RUNSTREAM 

DISPLAY  7.  CONDITION-LEVEL  CORRELATION  DISPLAY 


06-07-90  13:49:47 
PAGE  1 


STIMULUS 


ALL  STIM. 
Z-SCORE 

ALL  STIM. 
LSR 

BL  STIMULI 
Z-SCORE 

BL  STIMULI 
LSR 


RATING 


TRANSFORMS  BASED  ON  OBSERVERS'  RESPONSES  TO 
1)  ALL  INCLUDED  STIMULI    2)  BASELINE  STIMULI 

Z  LSR  Z  LSR  SBE 


.989 


.946 


.959 


.923 


.984 


.991 


.970 


.999 


.998 


.994 


SBE 


.976 


.932 


.853 


.874 


.816 


24 


SAMPLE  RUNSTREAM 

DISPLAY  8.  SCATTERGRAM  OF  TWO  OUTPUT  MEASURES  (CORRELATION  LESS  THAN  .986) 
1.85  + 


06-07-90  13:49:47 
PAGE  1 

1 


1 


1.42  + 


.88  + 


BL  STIMULI 
Z-SCORE 


.33  + 


.21  + 


1  1 
1  1 


1  1  1 
1  1 


1 


-.81  +  1 
+  . . . . 
-48.629 


1    1  1 
1  1 


1 

1  1 


....  +  . 
-31.874 


 +  . 

-15.118 


. . .  +  +  . 

1.637  18.393 


 +  +  . 

35.148  51.904 


.  + . 
68.659 


....  +  . 
85.415 


 +  . 

102.170 


 + 

115.574 


SBE 


25 


SAMPLE  RUNSTREAM 

DISPLAY  8.  SCATTERGRAM  OF  TWO  OUTPUT  MEASURES  (CORRELATION  LESS  THAN  .986) 
7.36  + 


06-07-90  13:49:47 
PAGE  2 

1 


1 


6.83  + 


6.17 


BL  STIMULI 
LSR 


5.51 


1  1 


4.85- 


1  2 


1  2 
1 


1 


4.13  + 

+  . 

-48.629 


 +  . 

-31.874 


....  +  . 
-15.118 


.  . .  +  . 
1.637 


....  +  . 
18.393 


. .  .  .  +  +  . 

35.148  51.904 


. . . .  +  +  . 

68.659  85.415 


 +  . 

102.170 


 + 

115.574 


SBE 


26 


SAMPLE  RUNSTREAM 

DISPLAY  9.  Z-SCORES  AND  ASSOCIATED  STATISTICS 


06-07-90  13:49:47 


PAGE  1 


STD.  OBSERVERS: 


STIMULUS 

CONDITION 

MbAN 

MbU.  Dev. 

n  AMPC 

1 

o 
d. 

4 

c 
0 

O 

7 

o 

o 

■f  n 
1U 

■1  o 

MEAN 

.00 

.00 

.00 

.00 

.00 

.00 

.00 

.00 

.00 

MEDIAN 

STANDARD  DEVIATION 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

1.00 

RANGE 

3.4 

4.2 

3.7 

4.0 

5.0 

3.3 

3.3 

3.8 

3.9 

SKEWNESS 

.303 

.387 

.363 

.859 

1 .925 

-.069 

-.128 

.350 

.579 

CORRELATION  VW  GROUP 

.481 

.557 

.668 

.403 

.680 

.324 

.348 

.519 

.541 

SIGNIFICANCE  OF  R 

.004 

.001 

.000 

.014 

.000 

.040 

.030 

.002 

.001 

1 

BASELINE 

.32 

.58 

1.6 

-.34 

1.09 

.12 

-.48 

.04 

.45 

.96 

.08 

.95 

2 

BASELINE 

-.32 

.69 

2.3 

-.34 

-.49 

.12 

.32 

-.58 

-.37 

-.89 

.83 

-1 .51 

3 

BASELINE 

-.73 

.65 

1 .8 

-1.48 

-.49 

-1.74 

-1 .28 

.04 

-.78 

-.15 

-.68 

-.03 

4 

BASELINE 

-.17 

.91 

2.4 

-.34 

1 .09 

-.50 

-1 .28 

-.58 

-1 .20 

.96 

.83 

-.52 

6 

BASELINE 

-.12 

.62 

1.9 

-.34 

1 .09 

-.50 

-.48 

.04 

-.78 

.59 

-.68 

-.03 

7 

BASELINE 

-.31 

.45 

1.5 

-.34 

-1.02 

.12 

-.48 

.04 

-.37 

-.52 

-.68 

.46 

8 

BASELINE 

.71 

.89 

2.7 

.80 

2.15 

.75 

-.48 

-.58 

1 .28 

.22 

.83 

1 .44 

9 

BASELINE 

-.67 

.61 

1.7 

-1 .48 

-.49 

-.50 

-1 .28 

-.58 

.04 

.22 

-1 .44 

-.52 

10 

BASELINE 

2.07 

.76 

2.7 

1 .94 

1 .62 

1 .99 

1 .12 

3.80 

1 .69 

1 .70 

2.34 

2.43 

11 

MIXPRE 

-.52 

1.01 

2.8 

-1 .48 

.04 

-1 .12 

-1 .28 

-1 .21 

-.78 

1 .33 

.83 

-1.02 

12 

MIXPRE 

-.67 

.63 

1.8 

-1.48 

-.49 

.12 

.32 

-1.21 

-.37 

-1.26 

-.68 

-1.02 

14 

MIXPRE 

-.40 

.34 

1.1 

-.34 

-.49 

-.50 

-.48 

.04 

.04 

-.15 

-.68 

-1.02 

15 

MIXPRE 

.03 

.92 

2.5 

-.34 

-1 .02 

.75 

.32 

-.58 

.87 

.96 

.83 

-1 .51 

16 

MIXPRE 

.45 

1.15 

3.2 

.80 

-.49 

1 .37 

.32 

1 .92 

.87 

-1 .26 

1 .59 

-1 .02 

18 

MIXPRE 

-.10 

.61 

1.7 

.80 

-.49 

-.50 

-.48 

.67 

.45 

-.89 

.08 

-.52 

19 

MIXPRE 

-.63 

.90 

2.9 

-.34 

-.49 

-1.12 

-1.28 

-.58 

1.28 

-1.63 

-1.44 

-.03 

20 

MIXPRE 

-.24 

1.04 

3.2 

-.34 

-1 .02 

-1.12 

1 .92 

.04 

.87 

-1 .26 

-.68 

-.52 

21 

MIXPRE 

-.55 

.91 

3.4 

-.34 

-2.08 

-.50 

-.48 

-1 .21 

1 .28 

-.89 

-.68 

-.03 

22 

MIXPRE 

-.38 

.67 

2.3 

.80 

-1 .55 

-.50 

-.48 

-.58 

-.37 

-.52 

-.68 

.46 

23 

MIXPRE 

.39 

1.08 

3.4 

1 .94 

.56 

1.37 

-.48 

.67 

.04 

1 .33 

-1 .44 

-.52 

24 

MIXPRE 

1.64 

.62 

1.9 

.80 

2.15 

1.99 

2.72 

1.29 

1.28 

.96 

1.59 

1.93 

25 

MIXPOST 

-.20 

.97 

2.7 

-.34 

.56 

-1.12 

1.12 

.67 

-1.61 

.59 

-.68 

-1.02 

26 

MIXPOST 

.13 

.81 

2.3 

-.34 

-.49 

1.37 

.32 

.04 

-.37 

-.89 

.08 

1.44 

27 

MIXPOST 

-.31 

.90 

2.5 

-.34 

.04 

.75 

.32 

-.58 

-1.61 

-1.63 

.83 

-.52 

28 

MIXPOST 

.44 

.59 

2.0 

.80 

.04 

.75 

.32 

.04 

1.28 

.96 

-.68 

.46 

29 

MIXPOST 

.02 

1.01 

3.5 

.80 

.04 

-.50 

1.92 

-.58 

-1.61 

.59 

08 

-.52 

30 

MIXPOST 

.23 

.72 

2.3 

-1.48 

.56 

.75 

.32 

.04 

.87 

.22 

.83 

-.03 

31 

MIXPOST 

.04 

.80 

2.6 

-.34 

.04 

.12 

.32 

-.58 

-1.61 

.59 

.83 

95 

34 

MIXPOST 

.04 

.74 

2.1 

.80 

-.49 

-1.12 

-.48 

.04 

-.37 

.96 

.08 

.95 

35 

MIXPOST 

-.19 

1.08 

3.4 

1.94 

.56 

-1.12 

-.48 

.04 

-.37 

-1.26 

-1.44 

.46 

27 


SAMPLE  RUNSTREAM 

DISPLAY  10.  LEAST  SQUARES  RATINGS  AND  ASSOCIATED  STATISTICS 


06-07-90  13:49:47 
PAGE  1 


STD.  OBSERVERS: 


STIMULUS 

CONDITION 

MEAN  MED. 

DEV. 

RANGE 

1 

2 

4 

5 

6 

7 

8 

10 

12 

MEAN 

4.87 

4.87 

4.87 

4.87 

4.87 

4.87 

4.87 

4.87 

4.87 

MEDIAN 

STANDARD  DEVIATION 

.59 

.75 

.79 

.51 

.80 

.56 

.63 

.67 

.74 

RANGE 

2.0 

3.2 

3.0 

2.0 

4.0 

1.8 

2.1 

2.5 

2.9 

SKEWNESS 

.303 

.387 

.363 

.859 

1.925 

-.069 

-.128 

.350 

.579 

CORRELATION  W/  GROUP 

.488 

.565 

.658 

.410 

.671 

.336 

.365 

.528 

.544 

SIGNIFICANCE  OF  R 

.003 

.001 

.000 

.012 

.000 

.035 

.024 

.001 

.001 

1 

BASELINE 

5.11 

.39 

1.1 

4.67 

5.69 

4.97 

4.63 

4.90 

5.12 

5.47 

4.92 

5.57 

2 

BASELINE 

4.64 

.48 

1.7 

4.67 

4.50 

4.97 

5.03 

4.40 

4.66 

4.31 

5.42 

3.76 

3 

BASELINE 

4.40 

.45 

1.4 

3.99 

4.50 

3.49 

4.22 

4.90 

4.43 

4.78 

4.42 

4.85 

4 

BASELINE 

4.78 

.58 

1.5 

4.67 

5.69 

4.47 

4.22 

4.40 

4.20 

5.47 

5.42 

4.48 

6 

BASELINE 

4.81 

.42 

1.3 

4.67 

5.69 

4.47 

4.63 

4.90 

4.43 

5.24 

4.42 

4.85 

7 

BASELINE 

4.68 

.32 

1.1 

4.67 

4.10 

4.97 

4.63 

4.90 

4.66 

4.54 

4.42 

5.21 

8 

BASELINE 

5.36 

.64 

2.1 

5.34 

6.49 

5.46 

4.63 

4.40 

5.59 

5.01 

5.42 

5.94 

9 

BASELINE 

4.43 

.36 

1.1 

3.99 

4.50 

4.47 

4.22 

4.40 

4.89 

5.01 

3.91 

4.48 

10 

BASELINE 

6.31 

.70 

2.5 

6.02 

6.09 

6.45 

5.44 

7.91 

5.82 

5.94 

6.43 

6.66 

11 

MIXPRE 

4.52 

.67 

1.8 

3.99 

4.90 

3.98 

4.22 

3.90 

4.43 

5.71 

5.42 

4.12 

12 

MIXPRE 

4.41 

.42 

1.1 

3.99 

4.50 

4.97 

5.03 

3.90 

4.66 

4.08 

4.42 

4.12 

14 

MIXPRE 

4.60 

.25 

.8 

4.67 

4.50 

4.47 

4.63 

4.90 

4.89 

4.78 

4.42 

4.12 

15 

MIXPRE 

4.85 

.65 

1.7 

4.67 

4.10 

5.46 

5.03 

4.40 

5.35 

5.47 

5.42 

3.76 

16 

MIXPRE 

5.19 

.83 

2.3 

5.34 

4.50 

5.96 

5.03 

6.40 

5.35 

4.08 

5.93 

4.12 

18 

MIXPRE 

4.80 

.41 

1.1 

5.34 

4.50 

4.47 

4.63 

5.40 

5.12 

4.31 

4.92 

4.48 

19 

MIXPRE 

4.44 

.55 

1.7 

4.67 

4.50 

3.98 

4.22 

4.40 

5.59 

3.85 

3.91 

4.85 

20 

MIXPRE 

4.65 

.63 

1.9 

4.67 

4.10 

3.98 

5.85 

4.90 

5.35 

4.08 

4.42 

4.48 

21 

MIXPRE 

4.46 

.63 

2.3 

4.67 

3.30 

4.47 

4.63 

3.90 

5.59 

4.31 

4.42 

4.85 

22 

MIXPRE 

4.60 

.48 

1.6 

5.34 

3.70 

4.47 

4.63 

4.40 

4.66 

4.54 

4.42 

5.21 

23 

MIXPRE 

5.14 

.72 

2.1 

6.02 

5.30 

5.96 

4.63 

5.40 

4.89 

5.71 

3.91 

4.48 

24 

MIXPRE 

5.97 

.43 

1.1 

5.34 

6.49 

6.45 

6.26 

5.90 

5.59 

5.47 

5.93 

6.30 

25 

MIXPOST 

4.73 

.63 

1.5 

4.67 

5.30 

3.98 

5.44 

5.40 

3.97 

5.24 

4.42 

4.12 

26 

MIXPOST 

4.99 

.59 

1.6 

4.67 

4.50 

5.96 

5.03 

4.90 

4.66 

4.31 

4.92 

5.94 

27 

MIXPOST 

4.69 

.58 

1.6 

4.67 

4.90 

5.46 

5.03 

4.40 

3.97 

3.85 

5.42 

4.48 

28 

MIXPOST 

5.15 

.37 

1.2 

5.34 

4.90 

5.46 

5.03 

4.90 

5.59 

5.47 

4.42 

5.21 

29 

MIXPOST 

4.84 

.58 

1.9 

5.34 

4.90 

4.47 

5.85 

4.40 

3.97 

5.24 

4.92 

4.48 

30 

MIXPOST 

5.04 

.45 

1.5 

3.99 

5.30 

5.46 

5.03 

4.90 

5.35 

5.01 

5.42 

4.85 

31 

MIXPOST 

4.91 

.50 

1.6 

4.67 

4.90 

4.97 

5.03 

4.40 

3.97 

5.24 

5.42 

5.57 

34 

MIXPOST 

4.89 

.51 

1.6 

5.34 

4.50 

3.98 

4.63 

4.90 

4.66 

5.47 

4.92 

5.57 

35 

MIXPOST 

4.74 

.70 

2.1 

6.02 

5.30 

3.98 

4.63 

4.90 

4.66 

4.08 

3.91 

5.21 

28 


SAMPLE  RUNSTREAM 

DISPLAY  11.  BASELINE-ADJUSTED  Z-SCORES  AND  ASSOCIATED  STATISTICS 


06-07-90  13:49:47 
PAGE  1 


STD.  OBSERVERS: 


STIMULUS 

CONDITION 

MEAN  MED 

DEV. 

RANGE 

1 

2 

4 

5 

5 

7 

8 

10 

12 

MEAN 

44 

-U  I 

fin 

-OU 

1  t 

— .  1  o 

nn 

-HO 

14 

— .^o 

MEDIAN 

STANDARD  DEVIATION 

.94 

.88 

.98 

1.25 

.72 

1.02 

1.24 

.86 

.85 

RANGE 

Q  9 
O.t 

1  7 

Q  7 
o.  / 

o.u 

O.D 

^  A 

A  1 

o.o 

1 1 
o.o 

SKEWNESS 

^n^ 
.ouo 

1R7 
.Oo  / 

QfiQ 

ftRQ 

.ooy 

— .uoy 

1  9ft 

OCA 
-OOU 

C7Q 

CORRELATION  W/  GROUP 

Afift 
.H  DO 

Rfin 

.ODD 

.ooy 

-D  /  D 

^n^ 
.ouo 

-OOU 

*^97 

Cill 

.ooo 

SIGNIFICANCE  OF  R 

.UUD 

nm 

.UU  I 

nnn 

m  7 

.U  I  f 

nnn 
.uuu 

n^i 

■  UO  I 

n<i7 
.uo/ 

nm 

nni 

BASELINE 

.24 

.34 

.9 

-.12 

.52 

.14 

.00 

-.10 

.47 

.76 

-.07 

.56 

2 

BASELINE 

ir 

—  .OO 

R7 
■  Of 

0  R 
d.O 

19 
— .  I  c. 

ftft 

14 

1  on 
I  .uu 

- .OO 

op 
— .oo 

—  I  -OO 

CO 

.DO 

—  I  .oo 

o 
O 

DACCI  IMC 

DAbbLINb 

H1 

A7 

1  fi 

-1  9f> 

—  ftft 
— .oo 

—  1  RQ 

—  1  .057 

—1  nn 

—  I  .uu 

-  1  n 

— .  1  u 

fin 

.OU 

fi1 

79 

9fl 
—.do 

4 

BASELINE 

-  94 

79 

O  Ci 

—  19 

c.9 

-  47 

-1  nn 

—  1  -UU 

— .oo 

-1  99 

7fi 
.  /  D 

CO 

.oo 

—  7n 

—  .  fU 

6 

BASELINE 

_  1  Q 

AA 

I  .o 

—  1  9 
— .  I  c 

t\0 

47 

nn 

-UU 

—  1  n 

— .  I  u 

fin 

.OU 

Q1 
.O  I 

79 

9ft 
— .^O 

7 

BASELINE 

-.38 

.54 

1.5 

-.12 

-1.35 

.14 

.00 

-.10 

-.38 

-1.07 

-.72 

.14 

8 

BASELINE 

fiQ 

9  n 

£.U 

QR 

.570 

1  AR 

74 

nn 

-UU 

— .oo 

1  11 

I  .O  I 

1  ^ 

— .  I  o 

CO 

.oo 

Q7 

y 

DACCI  IMC 

-  7fl 
— .  i  \J 

A7 

1  A 

1  .*T 

-1  9D 

-  ftft 
— -OO 

-  47 

—1  nn 

— .oo 

n^ 

.uo 

— .  I  o 

—  1  17 

—  1  .0/ 

_  7n 
— . /u 

10 

BASELINE 

1  rr 

AO 

1  fi 

9  f)4 

Qft 

1  QR 

9  nn 

9  fin 

1  74 

1  ftft 

I  .DO 

1  fiR 
I  -OO 

1  fi1 

l  .O  1 

11 

MIXPRE 

R1 
.oo 

.OD 

9  4 

1  9f1 
—  1  .^u 

—  AA 

-1  nfl 
—  I  .UO 

1  nn 

1  nn 

l  .uu 

fin 

— .OU 

1  99 

.OO 

111 

12 

MIXPRE 

-.68 

.86 

3.0 

-1.20 

-.88 

.14 

1.00 

-1.00 

-.38 

-1.99 

-.72 

-1.11 

14 

MIXPRE 

AA 

AO 
.^d 

I  .d 

1  9 

Rfi 
— .OO 

47 

nn 
.uu 

1  n 
- .  I  u 

.UO 

fi1 

— .D  I 

79 

—  1 .  1  I 

15 

MIXPRE 

.uo 

Qfi 

.yo 

O  *\ 
d.O 

1  9 

1  IR 
—  I  .OO 

74 

1  nn 

I  .uu 

— .oo 

ftQ 

.oy 

7fi 

.oo 

1  CO 

—  1 .00 

16 

MIXPRE 

^n 

•OU 

1  OR 

O.O 

QR 

fifi 
— .oo 

1  IR 
1  -OO 

1  nn 

I  .uu 

1  OR 

ftQ 

.oy 

1  QQ 

—  i  .yy 

1  91 
1  .^O 

111 
—  I .  I  I 

18 

MIXPRE 

91 

Qfi 

— .OO 

47 

nn 
.uu 

.OO 

47 

—  I  .oo 

n7 

7n 

19 

MIXPRE 

-.71 

1.02 

3.8 

-.12 

-.88 

-1.08 

-1.00 

-.55 

1.31 

-2.45 

-1.37 

-.28 

20 

MIXPRE 

94 

1  A7 

o.u 

1 9 

1  QR 

1  Oft 
I  .uo 

o.uu 

1  n 

.oy 

1  QQ 

—  i  .yy 

79 
—.id. 

7n 

—.  /V 

21 

MIXPRE 

—  R7 

.Of 

1  fi1 

Q  fi 

O.O 

19 

9  9ft 

A7 

nn 
.uu 

1  nn 
—  I  .uu 

1 .0  l 

—  I  .oo 

79 

9ft 

—.do 

22 

MIXPRE 

41 
—  .'to 

7ft 

.  /  O 

9  ft 

Qfi 

1  ft1 
—  I  .O  I 

47 

nn 
.uu 

RR 

—  .00 

Qft 
— .Oo 

1  n7 

"TO 

1  A 

23 

MIXPRE 

.OO 

I  .UD 

O.H 

9  f)4 

n^ 

1  Q^ 

nn 
.uu 

■  OO 

.uo 

1  99 

1  Q7 
—  I  .0/ 

7n 
— .  /u 

24 

MIXPRE 

1.54 

.99 

3.2 

.96 

1.45 

1.96 

4.00 

.80 

1.31 

.76 

1.23 

1.39 

25 

MIXPOST 

-.22 

1.09 

3.6 

-.12 

.05 

-1.08 

2.00 

.35 

-1.64 

.31 

-.72 

-1.11 

26 

MIXPOST 

.03 

.94 

2.9 

-.12 

-.88 

1.35 

1.00 

-.10 

-.38 

-1.53 

-.07 

.97 

27 

MIXPOST 

-.39 

1.13 

3.4 

-.12 

-.41 

.74 

1.00 

-.55 

-1.64 

-2.45 

.58 

-.70 

28 

MIXPOST 

.41 

.71 

2.0 

.96 

-.41 

.74 

1.00 

-.10 

1.31 

.76 

-.72 

.14 

29 

MIXPOST 

.05 

1.32 

4.6 

.96 

-.41 

-.47 

3.00 

-.55 

-1.64 

.31 

-.07 

-.70 

30 

MIXPOST 

.17 

.70 

2.2 

-1.20 

.05 

.74 

1.00 

-.10 

.89 

-.15 

.58 

-.28 

31 

MIXPOST 

-.02 

.79 

2.6 

-.12 

-.41 

.14 

1.00 

-.55 

-1.64 

.31 

.58 

.56 

34 

MIXPOST 

-.03 

.70 

2.0 

.96 

-.88 

-1.08 

.00 

-.10 

-.38 

.76 

-.07 

.56 

35 

MIXPOST 

-.30 

1.15 

4.0 

2.04 

.05 

-1.08 

.00 

-.10 

-.38 

-1.99 

-1.37 

.14 

29 


SAMPLE  RUNSTREAM 

DISPLAY  12.  BASELINE-ADJUSTED  LEAST  SQUARES  RATINGS  AND  ASSOCIATED  STATISTICS 


06-07-90  13:49:47 
PAGE  1 


STD.  OBSERVERS: 


STIMULUS 

CONDITION 

MEAN  MED. 

DEV. 

RANGE 

1 

2 

4 

5 

6 

7 

8 

10 

12 

MEAN 

5.39 

4.62 

5.13 

5.75 

4.95 

5.12 

4.64 

4.95 

4.79 

MEDIAN 

STANDARD  DEVIATION 

1.30 

.97 

1.27 

1.34 

.87 

1.28 

1.36 

1.00 

1.10 

RANGE 

4.5 

4.1 

4.7 

5.3 

4.3 

4.2 

4.5 

3.8 

4.3 

SKEWNESS 

.303 

.387 

.363 

.859 

1.925 

-.069 

-.128 

.350 

.579 

CORRELATION  W/  GROUP 

.473 

.555 

.665 

.393 

.683 

.319 

.326 

.517 

.538 

SIGNIFICANCE  OF  R 

.004 

.001 

.000 

.016 

.000 

.043 

.039 

.002 

.001 

1 

BASELINE 

5.39 

.40 

1.0 

4.95 

5.68 

5.29 

5.11 

4.99 

5.70 

5.95 

5.03 

5.83 

2 

BASELINE 

4.67 

1.01 

3.1 

4.95 

4.14 

5.29 

6.18 

4.45 

4.64 

3.43 

5.79 

3.13 

3 

BASELINE 

4.13 

.63 

2.1 

3.46 

4.14 

2.92 

4.04 

4.99 

4.12 

4.44 

4.27 

4.75 

4 

BASELINE 

4.79 

.84 

2.4 

4.95 

5.68 

4.50 

4.04 

4.45 

3.59 

5.95 

5.79 

4.21 

6 

BASELINE 

4.87 

.52 

1.6 

4.95 

5.68 

4.50 

5.11 

4.99 

4.12 

5.45 

4.27 

4.75 

7 

BASELINE 

4.68 

.61 

1.7 

4.95 

3.62 

5.29 

5.11 

4.99 

4.64 

3.94 

4.27 

5.29 

8 

BASELINE 

5.85 

.83 

2.3 

6.43 

6.71 

6.08 

5.11 

4.45 

6.75 

4.94 

5.79 

6.37 

9 

BASELINE 

4.27 

.58 

1.7 

3.46 

4.14 

4.50 

4.04 

4.45 

5.17 

4.94 

3.51 

4.21 

10 

BASELINE 

7.36 

.58 

2.0 

7.91 

6.20 

7.65 

7.25 

8.24 

7.28 

6.96 

7.30 

7.46 

11 

MIXPRE 

4.42 

1.03 

3.0 

3.46 

4.65 

3.71 

4.04 

3.91 

4.12 

6.45 

5.79 

3.67 

12 

MIXPRE 

4.28 

.99 

3.2 

3.46 

4.14 

5.29 

6.18 

3.91 

4.64 

2.93 

4.27 

3.67 

14 

MIXPRE 

4.58 

.51 

1.5 

4.95 

4.14 

4.50 

5.11 

4.99 

5.17 

4.44 

4.27 

3.67 

15 

MIXPRE 

5.15 

1.18 

3.1 

4.95 

3.62 

6.08 

6.18 

4.45 

6.22 

5.95 

5.79 

3.13 

16 

MIXPRE 

5.51 

1.49 

3.9 

6.43 

4.14 

6.86 

6.18 

6.62 

6.22 

2.93 

6.54 

3.67 

18 

MIXPRE 

4.90 

.92 

3.0 

6.43 

4.14 

4.50 

5.11 

5.53 

5.70 

3.43 

5.03 

4.21 

19 

MIXPRE 

4.30 

1.18 

4.3 

4.95 

4.14 

3.71 

4.04 

4.45 

6.75 

2.43 

3.51 

4.75 

20 

MIXPRE 

4.80 

1.63 

5.4 

4.95 

3.62 

3.71 

8.32 

4.99 

6.22 

2.93 

4.27 

4.21 

21 

MIXPRE 

4.47 

1.17 

4.2 

4.95 

2.59 

4.50 

5.11 

3.91 

6.75 

3.43 

4.27 

4.75 

22 

MIXPRE 

4.64 

.93 

3.3 

6.43 

3.11 

4.50 

5.11 

4.45 

4.64 

3.94 

4.27 

5.29 

23 

MIXPRE 

5.55 

1.35 

4.4 

7.91 

5.17 

6.86 

5.11 

5.53 

5.17 

6.45 

3.51 

4.21 

24 

MIXPRE 

6.94 

1.05 

3.4 

6.43 

6.71 

7.65 

9.39 

6.07 

6.75 

5.95 

6.54 

6.91 

25 

MIXPOST 

4.78 

1.27 

4.2 

4.95 

5.17 

3.71 

7.25 

5.53 

3.06 

5.45 

4.27 

3.67 

26 

MIXPOST 

5.18 

1.11 

3.4 

4.95 

4.14 

6.86 

6.18 

4.99 

4.64 

3.43 

5.03 

6.37 

27 

MIXPOST 

4.64 

1.30 

3.8 

4.95 

4.65 

6.08 

6.18 

4.45 

3.06 

2.43 

5.79 

4.21 

28 

MIXPOST 

5.62 

.85 

2.5 

6.43 

4.65 

6.08 

6.18 

4.99 

6.75 

5.95 

4.27 

5.29 

29 

MIXPOST 

5.12 

1.51 

5.3 

6.43 

4.65 

4.50 

8.32 

4.45 

3.06 

5.45 

5.03 

4.21 

30 

MIXPOST 

5.29 

.89 

2.8 

3.46 

5.17 

6.08 

6.18 

4.99 

6.22 

4.94 

5.79 

4.75 

31 

MIXPOST 

5.07 

.94 

3.1 

4.95 

4.65 

5.29 

6.18 

4.45 

3.06 

5.45 

5.79 

5.83 

34 

MIXPOST 

5.09 

.87 

2.7 

6.43 

4.14 

3.71 

5.11 

4.99 

4.64 

5.95 

5.03 

5.83 

35 

MIXPOST 

4.81 

1.44 

5.0 

7.91 

5.17 

3.71 

5.11 

4.99 

4.64 

2.93 

3.51 

5.29 

30 


SAMPLE  RUNSTREAM 

DISPLAY  13.  PRINCIPAL  COMPONENT  ANALYSIS:  EIGENVALUES 


06-07-90  13:49:47 
PAGE  1 


EIGENVALUES: 

3.634           1.262  1.083  .8305  .5924  .4970  .4131  .3950  .2926 

PERCENT  OF  VARIANCE: 

40.38           14.02  12.04  9.228  6.582  5.522  4.590  4.389  3.251 

SAMPLE  RUNSTREAM  06-07-90  13:49:47 

DISPLAY  14.  PRINCIPAL  COMPONENT  ANALYSIS:  COMPONENT  LOADINGS  PAGE  1 


OBSERVER 

COMPONENT 

COMPONENT 

COMPONENT 

1 

2 

3 

1 

.613  * 

-.271 

.385 

2 

.678  * 

.503 

.146 

4 

.775  * 

-.167 

-.275 

5 

.540 

-.168 

-.556  * 

6 

.785  * 

-.184 

.075 

7 

.451 

-.588  * 

.208 

8 

.473 

.677  * 

.198 

10 

.652  * 

.204 

-.521 

12 

.662  * 

.000 

.411 

*  OBSERVER'S  STRONGEST  LOADING 

SAMPLE  RUNSTREAM 

DISPLAY  15.  PRINCIPAL  COMPONENT  ANALYSIS:  COMPONENT  SCORES 


STIMULUS  COMPONENT  COMPONENT  COMPONENT 


1 

2 

3 

1 

.169 

.181 

.161 

2 

-.173 

-.056 

-.315  L 

3 

-.403  L 

.144 

.125 

4 

-.087 

.425  H 

.022 

6 

-.060 

.251 

.130 

7 

-.154 

-.106 

.057 

8 

.388 

.082 

.198 

9 

-.384  L 

.065 

.134 

10 

1.181  H 

-.048 

.100 

11 

-.316 

.426  H 

-.069 

12 

-.371 

-.078 

-.238 

14 

-.227 

-.044 

.021 

15 

-.021 

-.030 

-.248 

16 

.298 

-.353  L 

-.266 

18 

-.047 

-.211 

.063 

19 

-.362 

-.297 

.259 

20 

-.165 

-.342  L 

-.173 

21 

-.344 

-.340 

.055 

22 

-.216 

-.165 

.144 

23 

.220 

.012 

.281  H 

24 

.908  H 

.011 

-.164 

25 

-.110 

.241 

-.106 

26 

.111 

-.135 

-.081 

27 

-.116 

.008 

-.332  L 

28 

.213 

-.103 

.166 

29 

-.003 

.170 

-.203 

30 

.124 

.051 

-.195 

31 

.034 

.274 

-.115 

34 

.003 

.113 

.228 

35 

-.090 

-.150 

.360  H 

H  =  HIGHEST  TWO  STIMULI  ON  COMPONENT 
L  =  LOWEST  TWO  STIMULI  ON  COMPONENT 


06-07-90  13:49:47 
PAGE  1 


31 


SAMPLE  RUNSTREAM 


06-07-90  13:49:47 


DISPLAY  16.  PRINCIPAL  COMPONENT  ANALYSIS:  LEGEND  FOR  SCATTERGRAM  (NEXT  PAGE) 

OF  COMPONENT  LOADINGS  PAGE  1 

COMPONENT  COMPONENT 
OBSERVER  SYMBOL  1  2 

1  A  .613  -.271 

2  B  .678  .503 

4  C  .775  -.167 

5  D  .540  -.168 

6  E  .785  -.184 

7  F  .451  -.588 

8  G  .473  .677 
10  H  .652  .204 
12  I  .662  .000 

SAMPLE  RUNSTREAM  06-07-90  13:49:47 

DISPLAY  16.  PRINCIPAL  COMPONENT  ANALYSIS:  SCATTERGRAM  (LEGEND  ON  PREVIOUS  PAGE) 

OF  COMPONENT  LOADINGS  PAGE  2 

.78+  E 

C 


.73  + 


.66  + 


COMPONENT 

1 


.59  + 


.53  + 


.45 +  F 

+  +  +  +  +  +  +  +  +  +  

-.575  -.447  -.319  -.192  -.064  .064  .192  .320  .447  .575  .677 

COMPONENT 
2 


32 


SAMPLE  RUNSTREAM 


06-07-90  13:49:47 


DISPLAY  16.  PRINCIPAL  COMPONENT  ANALYSIS:  LEGEND  FOR  SCATTERGRAM  (NEXT  PAGE) 

OF  COMPONENT  LOADINGS  PAGE  3 

COMPONENT  COMPONENT 
OBSERVER  SYMBOL  1  3 

1  A  .613  .385 

2  B  .678  .146 

4  C  .775  -.275 

5  D  .540  -.556 

6  E  .785  .075 

7  F  .451  .208 

8  G  .473  .198 
10  H  .652  -.521 
12                       I                        .662  .411 


SAMPLE  RUNSTREAM 


06-07-90  13:49:47 


DISPLAY  16.  PRINCIPAL  COMPONENT  ANALYSIS:  SCATTERGRAM  (LEGEND  ON  PREVIOUS  PAGE) 
OF  COMPONENT  LOADINGS 


PAGE  4 


.78  + 


.73 


.66- 


COMPONENT 
1 


.59 


D 


.53  + 


.45+  F 

+  +  +  +  +  +  +  +  +  +  + 

-.546  -.448  -.351  -.253  -.156  -.058  .040  .137  .235  .333  .411 

COMPONENT 
3 


33 


SAMPLE  RUNSTREAM 

DISPLAY  17.  PRINCIPAL  COMPONENT  ANALYSIS:  LEGEND  FOR  SCATTERGRAM  (NEXT  PAGE) 
OF  COMPONENT  SCORES 

COMPONENT  COMPONENT 


WULUS 

SYMBOL 

»  '  1  IVI  U^y  U 

-| 

2 

1 

A 

.169 

.181 

2 

B 

-.1 73 

-.056 

3 

c 

-.403 

.144 

4 

D 

-.087 

.425 

6 

E 

-.060 

.251 

7 

F 

-.154 

-.106 

8 

G 

.388 

.082 

9 

H 

-.384 

.065 

10 

| 

1.181 

-.048 

1 1 

j 

-.316 

.426 

12 

K 

-.371 

-.078 

14 

L 

-.227 

-.044 

15 

M 

-.021 

-.030 

16 

N 

.298 

-.353 

18 

o 

-.047 

-.21 1 

19 

p 

-.362 

-.297 

20 

Q 

-.165 

-.342 

21 

R 

-.344 

-.340 

22 

S 

-.216 

-.165 

23 

T 

.220 

.012 

24 

U 

.908 

.011 

25 

V 

-.110 

.241 

26 

w 

.111 

-.135 

27 

X 

-.116 

.008 

28 

Y 

.213 

-.103 

29 

z 

-.003 

.170 

30 

a 

.124 

.051 

31 

b 

.034 

.274 

34 

c 

.003 

.113 

35 

d 

-.090 

-.150 

34 


SAMPLE  RUNSTREAM 


06-07-90  13:49:47 


DISPLAY  17.  PRINCIPAL  COMPONENT  ANALYSIS:  SCATTERGRAM  (LEGEND  ON  PREVIOUS  PAGE) 
OF  COMPONENT  SCORES 


PAGE  2 


1.18  + 


.92- 


.60 


COMPONENT 
1 


.28- 


W 


.05- 


.40  + 

+  . 
-.346 


.  .  .  +  . 
-.267 


.  .  +  . 
.188 


.  .  +  . 
.109 


.  .  + 
-.031 


.  .  +  . 
.048 


.  .  +  . 
.127 


.  .  +  . 
.206 


.  .  +  . 
.284 


. .  +  . 
.363 


.  .  + 
.426 


COMPONENT 

2 


35 


SAMPLE  RUNSTREAM 


06-07-90  13:49:47 


DISPLAY  17.  PRINCIPAL  COMPONENT  ANALYSIS:  LEGEND  FOR  SCATTERGRAM  (NEXT  PAGE) 

OF  COMPONENT  SCORES  PAGE  3 

COMPONENT  COMPONENT 


STIMULUS 

SYMBOL 

1 

3 

1 

A 

.169 

.161 

2 

B 

-.173 

-.315 

3 

C 

-.403 

.125 

4 

D 

-.087 

.022 

6 

E 

-.060 

.130 

7 

F 

-.154 

.057 

8 

G 

.388 

.198 

9 

H 

-.384 

.134 

10 

I 

1.181 

.100 

11 

J 

-.316 

-.069 

12 

K 

-.371 

-.238 

14 

L 

-.227 

.021 

15 

M 

-.021 

-.248 

16 

N 

.298 

-.266 

18 

O 

-.047 

.063 

19 

P 

-.362 

.259 

20 

Q 

-.165 

-.173 

21 

R 

-.344 

.055 

22 

S 

-.216 

.144 

23 

T 

.220 

.281 

24 

U 

.908 

-.164 

25 

V 

-.110 

-.106 

26 

w 

.111 

-.081 

27 

X 

-.116 

-.332 

28 

Y 

.213 

.166 

29 

z 

-.003 

-.203 

30 

a 

.124 

-.195 

31 

b 

.034 

-.115 

34 

c 

.003 

.228 

35 

d 

-.090 

.360 

36 


SAMPLE  RUNSTREAM 


06-07-90  13:49:47 


DISPLAY  17.  PRINCIPAL  COMPONENT  ANALYSIS:  SCATTERGRAM  (LEGEND  ON  PREVIOUS  PAGE) 

OF  COMPONENT  SCORES  PAGE  4 

1.18+  I 


.92  + 


.60  + 


COMPONENT 

1 


.28  + 

Y 

A 

a  W 


M  Z 

0  E 

.05+  D 
X  V 


Q 


K  HP 
.40+  C 

+  +  +  +  +  +  +  +  +  +  + 

-.325  -.255  -.185  -.115  -.045  .025  .095  .164  .234  .304  .360 

COMPONENT 

3 
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SAMPLE  RUNSTREAM 


06-07-90  13:49:47 


DISPLAY  18.  PRINCIPAL  COMPONENT  ANALYSIS:  CORR.  OF  COMPONENT  SCORES 
WITH  OTHER  MEASURES  OF  INTEREST 


PAGE  1 


RATING 

MEDIAN 

STD.  DEV. 
RAT. 

OAR 
SBE 

ALL  STIM. 
Z-SCORE 

ALL  STIM. 
LSR 

BL  STIMULI 
Z-SCORE 


COMPONENT 

1 

.981 
.912 

-.224 
.981 
.979 

.998 

.999 


COMPONENT 

2 

.063 
.093 

-.001 
.063 
.056 

.001 

.022 


COMPONENT 

3 

.080 
-.071 

-.081 
.080 
.100 

.013 

.024 


.993 


.018 


-.011 


38 


SAMPLE  RUNSTREAM  06-07-90  13:49:47 

DISPLAY  19.  ANALYSIS  BY  OBSERVERS.  PAGE  1 

STD.  DEV.  -  INDICATIONS  OF  NORMALITY  - 

CONDITION/  MEAN  MEDIAN  OF  SKEWNESS  KURTOSIS  MOD.  A-D 

OBSERVER  RATING  RATING  RATINGS  SBE  RATING  RATING  RATING 


LL  STIM. 

4.87 

4.7 

1.74 

-12.39 

.51 

-.068 

1.228 

OBS. 

1 

7.30 

7.0 

.88 

30.12 

.30 

-.668 

1.937 

OBS. 

2 

4.93 

4.5 

1.89 

-34.70 

.39 

-.250 

.883 

OBS. 

4 

5.80 

5.5 

1.61 

9.17 

.36 

-.905 

.859 

OBS. 

5 

4.60 

4.0 

1.25 

17.13 

.86 

.322 

1.536 

OBS. 

6 

2.93 

3.0 

1.60 

-29.20 

1.93 

4.782 

2.262 

OBS. 

7 

4.90 

4.5 

2.43 

-12.50 

-.07 

-1 .206 

.740 

OBS. 

8 

5.40 

6.0 

2.70 

-36.56 

-.13 

-1.423 

.924 

OBS. 

10 

3.90 

4.0 

1.32 

-23.49 

.35 

-.801 

1.221 

OBS. 

12 

4.07 

4.0 

2.03 

-31.49 

.58 

-.460 

.694 

ASELINE 

5.11 

5.0 

1.82 

.00 

.53 

-.681 

.655 

OBS. 

1 

7.11 

7.0 

.93 

.00 

.65 

-.542 

.844 

OBS. 

2 

5.89 

7.0 

2.15 

.00 

-.01 

-1.819 

.652 

OBS. 

4 

5.78 

6.0 

1.64 

.00 

.31 

-.388 

.490 

OBS. 

5 

4.00 

4.0 

1.00 

.00 

.67 

-.778 

.699 

OBS. 

6 

3.22 

3.0 

2.22 

.00 

1.87 

2.105 

1.957 

OBS. 

7 

4.89 

4.0 

2.37 

.00 

.52 

-1.316 

.399 

OBS. 

8 

6.33 

6.0 

2.18 

.00 

.07 

-1.263 

.173 

OBS. 

10 

4.11 

4.0 

1.54 

.00 

.39 

-1.087 

.452 

OBS. 

12 

4.67 

4.0 

2.40 

.00 

.30 

-1.044 

.225 

ALL  NON-BASELINE 
STIMUL 


4.77 


4.6 


1.69 


-17.47 


.31 


-.694 


.971 


OBS. 
OBS. 
OBS. 
OBS. 
OBS. 
OBS. 
OBS. 
OBS. 
OBS. 


1 

2 
4 
5 
6 
7 
8 
10 
12 


7.38 
4.52 
5.81 
4.86 
2.81 
4.90 
5.00 
3.81 
3.81 


7.0 
4.0 
5.0 
5.0 
3.0 
5.0 
5.0 
4.0 
3.0 


.86 
1.66 
1.63 
1.28 
1.29 
2.51 
2.85 
1.25 
1.86 


26.14 
-57.23 
7.28 

29.08 
-35.46 
-12.63 
-50.79 
-23.90 
-39.72 


.13 
.36 
.36 
.80 
.60 
.28 
.00 
.20 
.57 


-.796 
.913 
-1.339 
.035 
-.114 
-1.325 
-1.715 
-1.219 
-.687 


1.334 
.780 

1.024 

1.306 
.809 
.864 

1.064 
.897 
.663 


MIXPRE 


4.71 


4.3 


1.76 


-14.93 


.51 


-.783 


.751 


OBS. 
OBS. 
OBS. 
OBS. 
OBS. 
OBS. 
OBS. 
OBS. 
OBS. 


1 

2 
4 
5 
6 
7 
8 
10 
12 


7.33 
4.08 
5.83 
4.67 
2.83 
6.00 
4.67 
3.75 
3.25 


7.0 
4.0 
5.0 
4.0 
2.5 
6.5 
3.5 
3.0 
3.0 


.89 
2.02 
1.75 
1.50 
1.64 
1.76 
3.03 
1.42 
1.86 


12.77 
-66.50 
5.28 

24.25 
-20.60 

36.14 
-55.22 
-15.85 
-54.65 


.11 
.87 
.51 
.97 
.47 

-.28 
.36 
.40 

1.21 


-1.001 
.551 
-1.378 
-.190 
-1.148 
-1.552 
-1.706 
-1.442 
.821 


.746 
.733 
.772 
1.007 
.460 
.549 
.848 
.824 
.820 


MIXPOST 


4.84 


5.1 


1.44 


-10.74 


.03 


-1.340 


.689 


OBS. 
OBS. 
OBS. 
OBS. 
OBS. 
OBS. 
OBS. 
OBS. 
OBS. 


1 
2 
4 
5 
6 
7 
8 
10 
12 


7.44 
5.11 
5.78 
5.11 
2.78 
3.44 
5.44 
3.89 
4.56 


7.0 
5.0 
6.0 
5.0 
3.0 
4.0 
7.0 
4.0 
5.0 


.88 
.78 
1.56 
.93 
.67 
2.70 
2.70 
1.05 
1.67 


12.01 
-27.83 
-1.86 
39.10 
-17.54 
-51.26 
-34.82 
-8.30 
-6.15 


.15 
-.15 
-.03 
.65 
.18 
.46 
-.56 
-.38 
-.09 


-1 .044 
-1.536 
-1.858 
-.542 
-1.120 
-1.461 
-1.595 
-1.330 
-1.575 


.588 
.750 
.622 
.844 
1.008 
.775 
.813 
.518 
.279 
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Brown,  Thomas  C;  Daniel,  Terry  C;  Schroeder,  Herbert  W.;  Brink, 
Glen  E.  1990.  An  analysis  of  ratings:  a  guide  to  RMRATE.  Gen. 
Tech.  Rep.  RM-195.  Fort  Collins,  CO:  U.S.  Department  of  Agricul- 
ture, Forest  Service,  Rocky  Mountain  Forest  and  Range  Experiment 
Station.  40  p. 

This  report  describes  RMRATE,  a  computer  program  for  analyzing 
rating  judgments.  RMRATE  scales  ratings  using  several  scaling  pro- 
cedures, and  compares  the  resulting  scale  values.  The  scaling  proce- 
dures include  the  median  and  simple  mean,  standardized  values,  scale 
values  based  on  Thurstone's  Law  of  Categorical  Judgment,  and 
regression-based  values.  RMRATE  also  computes  reliability  statistics 
and  analyzes  the  ratings  using  principal  component  analysis  tech- 
niques to  assess  variation  among  judges.  RMRATE  should  be  useful 
to  practitioners  needing  to  summarize  or  analyze  rating  data,  and  to 
researchers  interested  in  comparing  and  evaluating  alternative  scaling 
methods. 

Keywords:  Ratings,  psychological  scaling,  reliability,  judgment 
scaling  model 
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U.S.  Department  of  Agriculture 
Forest  Service 

Rocky  Mountain  Forest  and 
Range  Experiment  Station 


The  Rocky  Mountain  Station  is  one  of  eight 
regional  experiment  stations,  plus  the  Forest 
Products  Laboratory  and  the  Washington  Office 
Staff,  that  make  up  the  Forest  Service  research 
organization. 

RESEARCH  FOCUS 

Research  programs  at  the  Rocky  Mountain 
Station  are  coordinated  with  area  universities  and 
with  other  institutions.  Many  studies  are 
conducted  on  a  cooperative  basis  to  accelerate 
solutions  to  problems  involving  range,  water, 
wildlife  and  fish  habitat,  human  and  community 
development,  timber,  recreation,  protection,  and 
multiresource  evaluation. 

RESEARCH  LOCATIONS 

Research  Work  Units  of  the  Rocky  Mountain 
Station  are  operated  in  cooperation  with 
universities  in  the  following  cities: 


Albuquerque,  New  Mexico 

Flagstaff,  Arizona 

Fort  Collins,  Colorado* 

Laramie,  Wyoming 

Lincoln,  Nebraska 

Rapid  City,  South  Dakota 

Tempe,  Arizona 


•Station  Headquarters:  240  W.  Prospect  St.,  Fort  Collins,  CO  80526 


